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CHAPTER  I 


INTRODUCTION 


Computer  based  Army  tactical  intelligence  systems  must  con- 
sider a wide  variety  of  message  types  involving  a multiplicity  of  content 
areas.  Moreover,  such  diversified  information  must  be  entered  accu- 
rately into  a data  information  bank  and  be  in  such  a form  that  the  infor- 
mation can' be  retrieved  quickly,  thoroughly,  and  accurately.  Messages 
received  for  entry  into  the  system  may  range  from  informal  reports  rela- 
tive to  the  morale  and  motivation  of  hostile  forces  to  "hard”  information 
about  troop  movements  and  armament.  In  order  that  s.uch  information 
will  be  entered  in  a systematic  manner,  a standard*  scheme  is  required. 
Such  a scheme  would  allow  not  only  quick  and  accurate  data  entry  but 
also  data  recovery.  The  present  study  had  the  goal  of  developing  a sys- 
tem for  the  classification  and  retrieval  of  battlefield  messages  based  on 
linguistic  principles  and  language  perceptions  of  the  user. 


Objective 

The  specific  purposes  of  the  present  program  were  to  develop  an 
approach  to  the  organization  of  the  linguistic  data  which  are  employed  in 
an  Army  battlefield  oriented,  computer  -based  iiitelligence  system.  To 
achieve  this  goal,  a sequential  set  of  steps  was  involved;  (1)  content  and 
factor  analysis  (multidimensional  scaling)  to  develop  a basis  for  organiz- 
ing and  classifying  such  linguistic  data,  (2)  development  of  a taxonomy 
for  categorizing  such  data,  (3)  test  of  the  utility  and  reliability  of  the 
organizational/classificatory  taxonomy,  and  (4)  preliminary  definition 
of  a system  for  automatic  classification  of  battlefield  messages. 


Linguistic  Logic  of  Approach 

The  basic  linguistic  logic  of  the  present  approach  was  that  the 
developed  classification  method,  if  it  is  to  be  at  all  useful,  must  be  based 
on  a substrate  which  mimics  intelligence  analyst  language  processing  and 
understanding  in  the  tactical  intelligence  situation.  We  hold  that  schemes 
which  are  not  compatible  with  these  processes  will  fail  to  possess  utility 
regardless  of  their  merits  in  other  regards. 


Additional!}-,  if  the  same  information  appears  in  two  different 
forms,  as  in:  “The  enemv  was  seen  building  up  troops,"  and  "An  ene- 
my troup  buildup  was  seen,"  the  scheme  must  recognize  the  redundan- 
cy. More  importantly,  in  a message  such  as  "l  saw  enemy  troup  move- 
ments, and  a captured  enemy  soldier  reports  that  there  are  additional 
troops  at  a distance  of  10  kilometers,  " the  method  must  permit  a differ- 
entiation between  what  the  sender  of  the  message  was  able  to  report  as 
being  true,  and  what  was  found  out  from  a possibly  unreliable  source. 

The  message  might  then  be  classified  either  on  the  basis  of  the  reliable 
or  on  the  basis  of  the  unreliable  information,  or  on  both. 

• 

A method  which  depends  on  picking  out  key  words  or  phrases  as 
a basis  for  classification,  on  the  other  hand,  cannot  make  such  a subtle 
distinction.  Since  the  message  contains  references  to  a captured  soldier, 
any  classificatory  scheme  which  uses  a "captured  soldier"  key  word  will 
result  in  the  entire  message  being  associated  with  that  keyword.  Similarly, 
if  the  classificatory  scheme  does  not  use  a "captured  soldier"  keyword, 
then  the  fact  is  lost  that  a captured  soldier  is  a source  for  some  of  the  in- 
formation in  the  message. 

The  requirement  for  such  subtle  distinctions  is  even  more  impor- 
tant in  the  present  context  because  there  is  a greater  permanence  to 
•the  classification  scheme  than  heretofore.  A description  of  the  elements 
which  are  emerging  in  a particular  situation  is  perhaps  enough  to  classify 
messages  when  the  user  is  fully  aware  of  the  total  situational  context.  How- 
ever, in  the  present  case,  this  intimate  familiarity  cannot  be  assumed,  and 
the  relationships  among  emerging  elements  becomes  crucial. 


Transformational  Grammar 


The  methodology  of  transformational  grammar  developed  by  Harris 
(1968)  provided  the  basis  for  a grammar  of  the  battlefield  situation.  The 
nature  of  the  grammar  arrived  at,  when  one  is  dealing  with  a natural  lang- 
uage like  English,  is  discussed  in  detail  in  Harris'  .Mathematical, Struc - 
tures  of  Language.  Similar  work  has  been  done  on  oclier  natural  lan- 
guages, most  noiaoly,  the  work  of  Gross  (1968)  on  French.  The  work  of 
Hiz  and  Joshi  (1968)  is  also  pertinent.  In  the  present  work,  these  same 
techniques  were  used  to  deal  with  the  battlefield  intelligence  situation. 

The  language  of  the  battlefield  situation  is  of  course  English,  but  it  differs 
from  ordinary  English  in  that  grammatical  forms  in  the  btittlefield  situa- 
tion would  not  be  considered  grammatical  in  ordinary  English.  For  e.xam- 
ple,  in  the  battlefield  situation,  the  report  "Enemy  buildup  between  0800  and 
0900"  is  acceptable.  In  ordinary  English,  this  is  not  an  acceptable  sentence. 


In  this  respect,  battlefield  Encrlish  is  similar  to  "telegraph  English.  " In 
addition,  battlefield  English  is  much  smaller  in  extent  than  ordinary  English. 
There  are  many  words  and  many  methods  of  combining  words  into  sentences 
which  are  acceptable  in  ordinary  English  that  would  never  be  encountered  in 
battlefield  English.  For  example,  one  would. not  expect  to  find  the  word  "tu- 
multuous" at  the  battlefield,  nor  the  form  in  which  the  object  of  the  sentence 
is  brought  to  the  beginning  of  the  sentence,  as  in  "This  I know.  " According- 
ly, in  a sense,  battlefield  English  can  be  considered  a sublanguage  of  English. 
Harris  dealt  with  sublanguages  as  follows; 

Thus  the  sublanguage  grammar  contains  rules  which  the  lang- 
uage violates  and  the  language  grammar  contains  rules  which 
the  sublanguage  never  meets.  It  follows  that  while  the 
sentences  of  such  science  object-languages  are  included  in 
the  language  as  a whole,  the  grammar  of  these  sublanguages 
intersects  (rather  than  is  included  in)  the  grammar  of  the 
language  as  a whole  (p.l55). 


According  to  the  Harris  theory,  every  language  is  characterized  by 
classes  of  operators  and  ker.nels,  such  that  each  sentence  can  be  written  as 
a product  of  some  number  of  them.  The  operator  and  kernel  classes  differ 
from  language  to  language.  Examples, of  kernel  forms  for  English  are; 


noun,  verb,  noun  The  boy  hits  the  girl. 

noun,  verb,  noun,  preposition,  noun  The  boy  throws  the  ball  to  the  men. 


The  operator  types  are: 

Word  expansion 

The  flower  is  very  pretty. 

Verb  operators 

I enjoy  going  to  parties. 

Sentence  operators 

I suggest  you  go  home. 

Connectives 

The  men  who  objected  called  the  police. 

Permutations 

This  I like. 

Zeroings  and  pro-wordings 

It  is  an  interesting  book. 

Morphophenomics  (which  give  the  appropriate  final  shapes  to  the 
resultants  of  the  other  operators) 
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One  has  various  options  available  for  dealing  with  a sublanguage 
of  Enc’lish.  One  of  the  possibilities  is  to  start  with  a grammar  of  English, 
and  make  additions  and  deletions  where  necessary.  A second  is  to  apply  a 
grammar  of  English  directlv  to  the  corpus  under  consideration.  A third  is 
to  use  the  same  principles  that  are  used  in  developing  a grammar  of  English, 
but  only  to  consider  the  facts  of  the  sublanguage  itself,  without  reference  to 
the  facts  of  the  entire  English  language.  It  was  believed  that  the  third  ap- 
proach would  arrive  at  a defensible  set  of  results  in  the  most  efficient  man- 
ner. Accordingly,  the  battlefield  sublanguage,  without  reference  to  the  total 
language,  formed  the  basis  for  the  present  work 


Psychometric  Considerations 

While  a formal  linguistic  analytic  system  will  tell  us  something  about 
the  global  foundation  of  the  language  involved,  it  will  tell  us  little  about  the 
perceptual  organization  of  the  language  in  the  mind  of  the  language  user.  A 
grammar  is  extrinsic  to  the  human;  the  structure  of  the  language  in  the  mind 
of  the  user  is  a separate  issue.  In  fact,  the  perceptual  structure  ^ ^ 
among  user  subpopulations.  To  derive  the  perceptual  structure  of  battlefield 
analysts  relative  to  battlefield  English,  multidimensional  scaling  technique;, 
were  employed  in  the  present  work. 

• Multidimensional  scaling  and  related  clustering  techniques  are  con- 
cerned primarilv  with  the  spatial  representation  of  relationships  among  be- 
havioral data- -in  our  case,  similarity  perceptions  by  experts  among  military 
field  information  statements. 

In  areas  where  the  variables  are  complex  and  the  dimensions  unknown 
or  doubtful,  it  is  appropriate  to  delineate  the  variables  through  multidimen- 
sional scaling  analysis,  rather  than  to  establish  the  dimensions  arbitrarily. 
Evidence  of  the  validity  of  the  multidimensional  method  has  been  cited,  es- 
pecially in  research  where  dimensions  are  well  established. 

The  utility  of  the  multidimensional  scaling  for  yielding  an  empirically 
sound  and  logical  basis  for  describing  a complex  has  been  demonstrated  in  a 
variety  of  contexts  including  attitudes  (Messick,  1954,  19o6;  Abelson.  19o4), 
personality  (Jackson,  Messick.  k-.  Solley,  1957).  jobs  (Reeb.  10o9),  civi  de- 
fense (Smith  Siegel.  1967),  and  display  evaluation  (Silver,  Landis.  & Jones, 
1965)  Naval  electronic  maintenance  (Schultz  & Siegel,  1964).  Air  borce  tasks 
(Siegel  Bergman,  1073),  and  marketing  analysis  (Green  ic  Carmone,  19  <0) 
Messick,  in  his  studies  on  color,  concluded  that  "since  multidimensional  scal- 
ing procedures  yielded  structures  which  correlated  highly  with  the  ijvised 
Munscll  system,  it  would  now  seem  reasonable  to  apply  those  procedures^for 
purposes  of  exploration  and  discovery  in  areas  of  unknown  dimensiona  ity 

[1956.  p.  374]. 
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While  it  is  recognized  that  the  methodological  issues  of  multidimen- 
siOfial  scaling  continue  to  remain  somewhat  fluid  (Coombs  et  al.  , 1970), 
the  inii>OFtance  of  these  theoretical  controversies  is  not  considered  to  be 
highly  relevant  to  the  present  study.  Actually,  anv  of  the  content  or  dis- 
tance methods  would  probably  result  in  a reasonable  and  usable  structure 
(Sjoeberg,  1975J, 

in  the  content  approach  (Ekman,  1965),  judges  provide  estimates 
©f  the  similarity  between  all  possible  pairs  of  the  stimuli  under  considera- 
tion, The  similarity  magnitude  estimates  are  made  along  a 0 to  100  scale, 
and  these  values  may  be  taken  as  correlation  coefficients.  A large  amount 
©f  Work  has  been  done  with  the  Ekman  similarity  analysis  technique  and  how 
it  Compares  with  other  multidimensional  scaling  techniques  since  Ekman 
first  introduced  the  technique  in  1954  (Andrews  6c  Ray,  1957;  Brown,  1967; 
C©©mbs,  1964;.  Ekman,  1954,  1965,  1969;  Kuennapas,  1966,  1967;  Pfeiffer, 
KUehhapas,  & Fastiggi,  1073;  Sjoeberg,  1975;  Stone  cc  Coles,  1971).  Ekman 
(1970)  concluded  that  such  content  methods  are  superior  to  distance  methods 
When  the  purpose  is  to  reveal  underlying  dimensionality  in  such  areas  as  col- 
©r  perception,  odors,  emotions,  and  personality  traits.  While  Kuennapas 
(1966>  1967)  found  that  some  mechanism  other  than  similarity  was  at  work 
in  the  perception  of  form  and  memory  of  form,  the  similarity  analysis  reveal- 
ed the  underlying  dimensionality.  In  a similar  vein,  Pfeiffer  et  al.  (1973), 

Who  studied  the  teaching  job,  found  support  for  the  content  approach  to  mul- 
tidimensional scaling. 

While  the  content  method  of  Ekman  (1965)  is  one  of  several  similar- 
ity methods  which  has  found  wide  application  and  support,  one  may  not  wish 
to  employ  the  method  in  its  original  form  where  bipolar  factors  are  sought, 

©b  in  the  case  where  one  is  interested  in  obtaining  the  perceptual  structure 
based  on  a single  subject.  For  such  applications,  the  Stone,  Coles,  and 
LUidem  (1970)  revision  of  the  Ekman  method  may  be  employed.  The  Stone, 
C&les,  and  Lindem  revision  treats  the  similarity  matrix  given  by  the  subject 
as  raw  data,  intercorrclates  the  similarity  estimates  in  the  matrix,  colum.n 
by  Column,  and  then  enfer.s  the  resulting  transformation,  a correlational 
similarity  matrix,  for  factor  analysis.  By  this  statistical  technique,  the 
®l^hy  judgments  of  the  paired  columns  are  converted  into  single  judgments 
J)osscssing  ratio  scale  properties.  After  all  columns  have  been  paired  two 
at  a time,  a symmetrical  matrix  of  true  correlation  coefficients  numberi.ng 
as  many  as  the  original  raw  similarity  matrix  results.  Accordinnly,  meet- 
ing the  strict  assumptions  of  traditional  factor  analysis  represents  no  nroo- 
lem  for  the  user  and,  similarly,  causes  no  restriction  on  the  interpretation 
of  the  results. 


Overview  of  Approach 

To  accomplish  the  goals  of  the  total  present  program,  a system- 
atic set  of  steps  was  followed.  These  steps  are  presenteo  in  Figure  1-1. 
First,  a set  of  military  field  messages  was  obtained.  These  messages 
were  linguistically  analyzed  to  yield  kernels  and  operators,  and  these 
kernels  and  operators  were  grouped  into  a set  of  logical  kernel  classes 
and  operator  classes.  Then  a sample  of  messages  representing  each 
kernel  and  each  operator  class  was  subjected  to  a set  of  multidimension- 
al scaling  analyses  to  yield  the  dimensions  (perceptual  factors)  involved. 
The  methods,  procedures,  and  results  of  these  steps  constitute  the  basis 
for  Chapter  II  of  the  present  report.  The  emergent  dimensions  were  next 
woven  into  a taxonomy  or  classification  scheme  into  which  military  field 
messages  can  be  cast.  Chapter  III  describes  the  taxonomy.  Then  the 
utility/  reliability  of  the  derived  taxonomy  was  tested  from  both  the  in- 
formation classification  and  the  information  extraction  points  of  view 
(Chapter  IV).  Finally,  the  taxonomy  was  verified  against  the  opinions 
of  experienced  intelligence  analysts,  and  a computer  oriented  system 
for  automatic  processing  of  intelligence  information  was  conceptualized 
(Chapter  V). 


Content 

Classes  of  Oper- 

Multidimensional 

Taxonomy 

Analysis 

ators  &,  Kernels 

Scaling  Analysis 

Preparation 

Validation 
(2  experiments) 


Revision 

System  for  Computer 
Implementation 


Figure  1-1.  Program  sequence. 
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CHAPTER  II 


MULTIDIAIEXSIONAL  SCALING  ANALYSIS 


Message  Set  and  Linguistic  Analysis 

. In  order  to  derive  a battlefield  oriented  classification  scheme,  a 
realistic  message  set  was  required.  A set  of  110  messages  was  provided 
by  the  U.  S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sci- 
ences. The  messages  were  part  of  a scenario  which  concerned  a worsen- 
ing cold -war  situation  on  the  border  between  East  and  West  Germany.  For 
the  most  part,  the  messages  contained  two  to  four  sentences  each.  All 
messages  contained  information  about  enemy  activities.  Accordingly, 
the  classification  scheme  emerging  from  the  present  work  may  be  limited 
to  enemy  activities.  Examples  of  typical  messages  in  the  set  of  110  are; 


82.  Imagery  indicates  the  presence  of  a large  storage 
area  northeast  of  Werda 

83.  Observation  of  enemy  activity  along  east  side  of 
Wiesse-Elster  River  indicates  preparation  for  lay 
ing  pontoons 


Each  message  was  then  analyzed  according  to  the  theory  of  transforma- 
tional grammar.  Thus,  corresponding  to  each  sentence,  a set  of  kernels  and 
operators  was  derived  in  such  a way  that  the  original  sentence  w-as  transforma- 
tionally derived  from  that  set.  For  example,  a sentence  taken  from  one  mes- 
sage was:  Friendly  civilians  report  that  two  aggressor  bns  located  in  Schoneck 
as  of  12  0900  Sen  have  oeen  having  difficulty  in  maintaining  discipline  within 
the  town.  A transfcrmauonal  decomposition  of  this  sentence  yields  the  kernel 
sentences:  aggressor  bns  maintain  discipline  and  aggressor  bns  are  located  in 
Schoneck.  The  operators  are  friendly  civilians  report;  as  of  12  0900  Sep;  within 
the  town;  and  have  bee.n  having  difficulty.  According  to  the  theory,  each  oper- 
ator when  appliea  to  any  of  tiie  sentences  to  which  it  is  truly  applicable  has  an 
effect  of  a specified  type.  For  example,  the  first  operator,  friendly  civilians 
report,  has  the  effect  of  making  a verb  phrase  complement  of  its  operand,  so 
that  the  new  transformed  sentence  has  friendly  civilians  as  its  subject  and  re- 
port as  its  verb.  For  the  operators  as  of  120000  Sep  and  widnin  the  town,  the 
sentences  on  which  they  act  merely  take  the  operators  as  adverbs.  For  the 
operator  have  been  having  difficulty,  the  result  is  to  make  the  verb  and  object 
of  the  original  sentence  into  a prepositional  phrase,  so  that  the  verb  of  the 
transformed  sentence  is  the  verb  of  the  operator. 
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In  addition,  there  is  an  adjunct  operator,  two,  which  can  be  placed 
before  the  phrase  afigressor  bns.  So  far,  forms  friendly  civilians  report 
that  two  aggressor  bns  have  been  having  difficulty  in  maintaining  discipline 
within  the  towii  and  aggressor  bns  are  located  in  Schoneck  as  of  12  0900  Sec 
have  been  accounted  for.  The  conjunction  of  these  two  forms,  using  the  oper- 
ator vdtmli  followed  by  an  operator  which  deletes  redundancies  (i.  e. , aggres- 
sion bns)  accounts  for  the  final  form  of  the  message. 

The  performance  of  a transformational  analysis  of  such  material 
is  not  a cut  and  dried  one.  Since  a sublanguage  of  English  and  not  the  en- 
tire language  was  involved,  decisions  were  required  at  each  step  as  to  the 
correct  way  to  analyze  a sentence  within  that  sublanguage.  For  example, 
in  the  entire  language,  the  phrase  aggressor  troops  must  be  analyzed  as  a 
transformed  sentence  like  troops  belong  to  aggressor,  but  i.n  the  sublanguage, 
the  question  has  to  be  asked  whether  the  phrase  should  be  analyzed  in  such  a 
fashion  or  remain  unanalyzed.  The  latter  alternative  is  the  most  reasonable 
one  when  the  phrase  friendly  troops  or  its  synonyms  are  .not  likely  to  appear 
in  the  language;  even  when  such  a phrase  does  appear,  leaving  aggressor 
troops  unanalyzed  might  be  worthwhile  for  simplicity's  sake. 

As  a result  of  the  transformational  analysis,  a set  of  kernels  and 
operators  sufficient  to  derive  each  of  the  sentences  of  the  message  set  was 
obtained.  The  total  set  contained  about  400  kernels  and  about  2 50  operators. 

Classification  (Content  Analysis)  of  Kernels  and  Operators 

The  next  step  involved  sorting  the  kernels  and  operators  into  content 
classes.  The  basis  for  the  content  analysis  was  that  two  kernels  or  opera- 
tors were  placed  into  the  same  class  only  if  some  meaningful  classifier  rela- 
tionship existed  between  the  two.  B’or  example,  one  class  consisted  of  all 
operators  which  were  of  the  adverbial  form  meaning  "in  the  vicinity  of  a lo- 
cation"; one  class  of  kernels  consisted  of  verbs  which  were  paraphrasable 
by  the  term  "defensive.  " Some  such  linguistic  justification  was  at  the  root 
of  each  of  the  classes  obtained.  The  kernels  were  grouped  into  25  classes. 
The  operators  were  grouped  into  23  classes.  The  25  kernel  classes  are 
presented  in  Table  2-1.  Table  2-2  presents  the  operator  classes. 


8 


Table  2-1 


Names  of  Kernel  Classes' 


Civilian-military  relations 

Contents  of  documents 

Actions  of  enemy 

Weapons,  military  information 

Sabotage,  infiltration  on  both  sides 

Sources  of  and  methods  of  obtaining  information  - 

Composition  by  name 

Information  about  officers 

Strategy,  tactics,  doctrine 

*'Is  not  known'" 

Markings  and  insignia 
Replacements 

Type,  characterization  of  weapons,  etc. 

Compositions  of  weapons  and  of  men 
Actual  and  threatened  contact 
Sources  of  intelligence 
Strength  of  units 
Movement 

Training,  competence,  etc. 

Composition  by  number  and  type  of  equipment  or  by  number  of  men 
Morale  and  health 

Characterizations  of  missions,  roles,  etc. 

Analysis 

Information  about  informers,  individual  soldiers,  etc. 

Location 


Table  2-2 


Names  of  Ooerator  Classes 


Aspectuals  - training 
Aspectuals  - intention 
Aspectuals  - other 
Adverbs 


Adverbs 

Adverbs 

Binaries 

Adverbs 

Adjuncts 

Adjuncts 

Adjuncts 

Adjuncts 


position  - vicinity 
position  - other 
position  - unlocalized 


• time 

■ on  location  adverbs 

• enemy  activities 

■ mechanical 

■ other 

Significance  raisers  or  reducers 
Adverbs  - force  changers 

Adverbs  - other 

Sentence  operators 


high  reliability 
low  reliability 
deserter 
civilian 

training,  capability 
intention 

significance  reducer 


Unit  membership 


Stimulus  Preparation  for  Multidimensional  Scaling 

The  multidimensional  scaling  model  requires  that  each  member 
of  the  set  to  be  factored  be  pair  compared,  relative  to  similarity,  with 
every  other  member.  These  comparative  judgments  form  the  basic  data 
set  for  the  factor  analysis  which  is  the  final  step  of  the  multidimensional 
scaling  analytic  procedure.  To  this  end,  one  representative  was  selected 
from  each  class  of  operators  and  kernels  and  the.n  one  additional  stimulus 
was  selected  from  the  14  classes  (7  operators  and  7 kernels)  containing 
the  most  stimuli.  This  procedure  yielded  a set  of  62  stimuli.  This  final 
stimulus  set  is  presented  as  Table  2-3. 
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Analvsts  and  Procoriuro 


To  obtain  similarity  estimates  between  pairs  of  militarv  intelligence 
statements,  as  required  by  the  Stone  et  al.  (1970)  method  of  multidimension- 
al similarity  analysis,  four  retired  military  personnel  with  extensive  oper- 
ational experience  were  employed  as  analysts.  The  military  and  intelligence 
experience  of  these  persons  is  presented  in  Table  2-4. 


Table  2 -4 

Military  Background  and  Experience  of  Analvsts 


Subject 

Rank* 

Years  of  ZVlilitarv 

Service  bv 

Tvoe 

Militarv  Intelligence 

Other 

J.  OlSi 

Me 

Lt.  Col. 

1 

21 

22 

M 

Capt. 

4 

0 

4 

K 

Capt. 

3 

1 

4 

S 

Lt.  Col. 

13 

3 

16 

* Rank  at  discharge 

Each  analyst  was  briefed  in  a separate  data  collection  session  by  an  Apolied 
Psychological  Services'  staff  member.  The  briefing  included:  (1)  an  explana- 
tion of  the  purposes  and  background  of  the  study,  (2)  a description  of  military 
intelligence  message  list  and  the  basis  for  its  development,  and  (2)  an  expla.na- 
tion  of  the  manner  in  which  the  similarity  data  would  be  collected.  A scale  of 
similarity  values  was  distributed  to  each  judge;  each  judge  used  the  scale  in 
making  magnitude  estimations  of  the  degree  of  similarity  between  all  pairs  of 
messages  on  a 0 to  100  per  cent  scale.  Since  a large  number  of  comparisons 
[(62  X 61)/2]  was  involved,  each  analyst  responded  to  one-third  the  required 
estimations  on  one  of  three  separate  data  collection  sessions.  Each  data  col- 
lection session  involved  about  three  hours  work.  The  analysts  were  paid  for 
their  participation. 
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Por  control  purposes,  the  stimulus  pairs  were  presented  to  one- 
-hstU  of  the  analysts  in  a random  order  which  was  the  reverse  of  the  random 
cSHf^er  of  the  stimuli  for  the  other  analysts.  To  check  on  intra-analyst  reli- 
ftb-iUty,  ten  pairs  of  stimuli  were  presented  to  each  of  the  analysts  twice- - 

in  the  form  in  which  the  stimuli  to  be  judged  were  in  the  order  A-B  and 
Aho  time  with  the  stimuli  in  the  order  B-A. 


Results 


The  reliability  check  stimuli  were  interspersed  randomlv  throughout 
^.he  stimulus  set.  The  product  moment  correlation  was  calculated  between 
paired  data  sets. for  each  of  the  four  analysts.  The  product  mome.nt  cor- 
^■^.lations  so  derived  were:  . 68,  .98,  .31,  and  . 64.  The  mean  (employing  z' 
^.f^-nsformation)  reliability  is  . 77y  These  data  suggest  reasonable  subject 
igPfl.si.stency  and  minimum  effect  due  to  the  directionality  of  the  comparisons 
involyed. 


Pactor  Analysis 

The  similarity  estimates  given  by  the  four  analysts  resulted  in  four 
Separate  similarity^  matrices  and  one  based  on  the  average  of  the  four.  Each 
JUiatrix  was  62  x 62  in  dimension.  In  accordance  whth  the  Stone,  Coles,  and 
J=‘i.hdem  (1970)  method,  the  columns  of  each  raw  similaritv  matrix  were  inter- 
^prrelated.  The  result  of  this  step  was  a product  moment  correlation  similar 
iiy  matrix  containing  the  same  number  of  elements  as  the  original.  Then, 
each  analyst's  correlation  similarity  matrix,  as  well  as  the  averaged  data 
matrix,  was  subjected  to  factor  analysis  by  the  principal  components  method 
ivjth  orthogonal  rotation  in  accordance  w-ith  the  Kaiser  developed  varimax 
eriterion  (Dixon.  1 973;  B AID  03AI,  Revised  9-1-65).  Communalities  were 
estimated  at  1.  00.  Factor  extraction  was  stopped  after  the  16th  factor. 

The  cumulative  percentage  of  unrotated  variance  accounted  for  in  each  anal- 
ysis is  presented  in  Table  2 -5.  Table  2 -5  indicates  that  for  the  individual  anal- 
yses, 78  to  82  per  cent  of  the  total  variance  was  accounted  for  by  the  16  fac- 
tor solution.  For  the  averaged  matrix.  86  per  cent  of  the  variance  was  ac- 
eouhted  for. 


" When  this  value  is  corrected  by  the  Spearman-Erown  technique,  the 
overall  reliability  becomes  .93. 
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Tabic  2-5 


Factor 

Cumulative  Unrotated  Percentage  of  VariEince 
Accounted  for  by  Each  Factor  Analysis 

Average 

Subject 

Me 

M 

K 

S 

1 

« 

20 

18 

16 

20 

23 

2 

33 

31 

28 

29 

37 

3 

44 

40 

36 

35 

46 

4 

51 

47 

43 

41 

52 

5 

57 

53 

48 

' 46 

57 

6 

61 

58 

53 

51 

61 

7 

66 

62 

56 

55 

64 

8 

69 

66 

60 

59 

67 

9 

73 

69 

63 

62 

71 

10 

75 

71 

66 

64 

73 

11 

78 

73 

69 

67 

75 

12 

80 

75 

72 

70 

77 

13 

81 

77 

74 

72 

78 

14 

83 

79 

76 

74 

80 

15 

85 

80 

78 

76 

82 

16 

86 

82 

79 

78 

83 

Agreement  Across  Analyses 

The  next  step  involved  assessing  the  agreement  or  "congruency"  of 
the  factors  across  the  individual  factor  analysis  for  each  of  the  four  analysts. 

If  the  four  separate  analyses  yielded  factors  which  are  common,  some  general- 
ity can  be  attributed  to  the  factors  and  confidence  can  be  placed  in  the  factor 
analysis  which  integrated  the  data  of  the  four  subjects.  If  little  or  no  agree- 
ment among  the  perceptual  structures  of  the  four  analysts  is  found,  then  a 
Structure  which  possesses  general  utility  (at  least  for  the  message  set  in- 
volved) cannot  be  held  to  have  been  accomplished  by  the  integrated  analysis. 

Accordingly,  each  rotated  factor  solution  was  compared  with  every 
Other  rotated  solution,  making  a total  of  six  comparisons  among  four  anal- 
ysts. The  comparison  was  accomplished  in  accordance  with  the  procedures 
developed  by  Kaiser,  Hunka,  and  Bianchini  (1971).  This  method _^seeks  the 
degree  of  agreement  in  factorial  structures  by  a cosine  rotation  method.  In 
each  of  The  studios  being  compared  two  at  a time,  the  geometric  configura- 
tion of  Ihe  factor  vectors,  in  the  space  ..'here  cosines  of  the  angles  between 
vectors  signify  correlations,  are  considered.  Accordingly,  the  cosine  of  the 
angle  between  any  two  vectors  may  be  interpreted  as  the  correlation  bctwei'n 
the  two  variables  (factors)  represented. 
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For  the  present  investigation,  the  rotated  factorial  structures  of 
the  four  analvsts  were  compared  taking  them  two  at  a time,  making  a total 
of  six  coi-nparisons.  Since  each  factor  analysis  resulted  in  16  factors,  a 
16  X 16  matrix  of  2 56  cosines  representing  the  relationship  of  all  rotated 
factors  of  one  analyst  with  all  rotated  factors  of  the  comparison  analyst 
resulted.  The  cosine  matrices  for  all  comparisons  are  presented  m Ta- 
bles 2-6  through  2-11. 

Since  the  16  x 16  cosine  matrices  constituted  excessive  detail,  an  ap- 
propriate logic  and  technique  summarizing  the  cosines  across  factors  was 
sought.  Butzkamin  and  Pfeiffer  (1973)  were  faced  with  a similar  problem 
in  their  cross-cultural  study  of  the  perceived  structure  of  teaching.  To  as 
sess  the  average  degree  of  agreement  of  factors  across  two  sam.ples  based 
upon  different  individuals,  the  following  index  of  agreement  was  employed: 


■where: 


Ai  = index  of  agreement 

C = the  largest  cosine  in  any  column/ row  of  the  cosine  matrix 
^ indicating  which  factor  from  one  study  shows  most  agree- 
ment with  the  corresponding  factor  from,  the  other  study. 

In  the  event  of  a controversy  over  the  identification  of  the 
column/ row  intersection,  an  additional  cosine  is  identified. 

C = the  remaining  smaller  cosines  from  the  same  colum.n/row 
as  above,  representing  factors  having  smaller  agreements. 

N = the  total  number  of  possible  non-corresponding  factors  in 

the  columns/ row  of  the  matrix;  30  in  the  present  case  where 
matrix  size  is  16  x 16. 


To  calculate  the  index,  first  the  row/ column  intersect  possessing  the 
highest  value  for  each  column  was  identified.  In  the  case  of  a 16x  16  m.atrix 
(without  controversies)  there  were  16  such  intersects.  Each  intersect  inoi- 
cated  the  two  factors  possessing  the  greatest  congruency.  Then  the  formum  ^ 
was  applied.  The  advantage  of  this  index  over  other  similar  indices,  such  as 
the  Fisher  z’  technique  of  Kaiser,  Hunka,  and  Bianchini  (19/1),  is  that  the 
present  index  considers  both  agreement  and  disagreement.  Other  indices  only 
consider  agreement.  The  value  of  Ai  can  range  from  zero  (no  agreement)  to 
one  (perfect  agreement). 


.Table  2 - 


.14  -0.29  -0.21  0.28  0.04  0.08 


Since  a number  of  controversies  were  involved  in  row/ column  in- 
tersect selection,  the  number  of  comparisons  involved  in  each  case  ranged 
from  17  to  21.  The  resultant  indices  of  agreement  are  presented  in  Tables 
2-12  through  2-17.  The  individual  values  presented  in  Tables  2-12  through 
2-17  are  all  reasonably  high. 

To  obtain  the  average  agreement  of  all  factors  of  each  analysis  with 
all  factors  of  each  other  analysis,  the  indices  of  agreement  (Ai)  were  aver- 
aged across  the  total  number  of  potentially  corresponding  factors,  i.  e.  , 
highest  Ai  values.  The  results  are  presented  in  Table  2-18.  The  average 
indices  of  agreement  presented  in  Table  2-18  ranged  from  . 71  to  . 83. 

These  data  suggest  strong  agreement  among  the  factorial  structures  of  the 
data  yielded  by  the  four  separate  analysts.  . 

When  the  analysts  are  ordered  in  terms  of  their  average  degree 
of  agreement  with  each  other  (Table  2-18),  the  results  indicate  the  follow- 
ing order  from  most  to  least  interjudge  agreement; 


Me  > M > K > S 


Accordingly,  analyst  Me  (the  analyst  with  the  most  general  military  and 
least  military  intelligence  experience)  may  be  viewed  as  having  a percep- 
tion of  intelligence  communications  most  common,  or  most  central,  to  the 
other  experts.  However,  it  is  noted  that  the  differences  among  the  anal- 
ysts are  small.  The  reason  for  this  indication  (if,  in  fact,  true)  is  not 
known.  It  may  be  due  to  the  common  Army  background  of  the  analysts. 


* Additionally  7.',  agreement  coefficients  were  calculated  using  the  method 
suggested  by  Kaiser,  Hunka,  ai>d  Bianchini  (1971) . Only  trivial  differences 
were  noted  between  the  results  indicated  by  the  two  approaches. 
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Table  2-12 


Agreement 

Factors 

(Ai)  Among  Potentially  Congruent 
Produced  by  Analysts  M and  Me 

M 

Factors  of 

Me 

Ai 

. 1 

13 

0.94 

2 

7 

0.87 

3 

6 

0.97 

4 

2 

0.94 

5 

9 

0.67 

6 

12 

0.78 

7 

8 

0.78 

8 

3 

0.82 

9 

9 

0.89 

10 

10 

0.95 

11 

4 

0.82 

12 

5 

0.67 

13 

1 1 

0.89 

14 

15 

0.80 

15 

1 

0.73 

16 

16 

0.88 

5 

14 

0.60 

Mean  0.82 
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Table  2-13 


Agreement 

Factors 

(Ai)  Among  Potentially  Congruent 
Produced  by  Analysts  Me  and  K 

Me 

Factors  of 

K 

Al 

1 

4 

0.89 

2 

2 

0.95 

3 

• 

5 

0.74 

4 

1 1 

0.86 

5 

8 

0.75 

6 

6 

0.94 

7 

16 

0.74 

8 

13 

0.56 

9 

10 

0.92 

10 

3 

0.89 

1 1 

3 

0.57 

12 

8 

0.60 

13 

14 

0.69 

14 

12 

0.91 

15 

13 

0.63 

16 

7 

0.B2 

3 

1 

0.71 

13 

9 

0.66 

5 

15 

0.61 

Mean 


0,76 


Table  2-14 


Affeemon t 
factors 

Table  2-14 

(Ai)  Among  Potentially 
Produced  by  Analysts  M 

Congruent 
and  K 

M 

Factors  of 

K 

Ai 

1 

12 

0.58 

2 

1 

0.76 

2 

6 

0.78 

• 

4 

2 

0.78 

% 

1 

0.70 

e 

8 

0.85 

7 

13 

0.69 

g 

14 

0.81 

S 

10 

0.89 

10 

3 

0.79 

11 

11 

0.87 

12 

12 

0.57 

13 

9 

0.67 

14 

7 

0.65 

IS 

16 

0.67 

16 

IS 

0.79 

4 

4 

0.46 

13 

5 

0.48 

M^an  0.71 
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Table  2-15 


7 


Agreement 

Factors 

(Ai)  Among  Potentially 
Produced  by  Analysts 

Congruent 
Me  and  S 

Factors  of 

Me 

S , 

Ai 

1 

2 

0.73 

2 

4 

0.88 

3 

1 

0.86 

4 

14 

0.55 

5 

8 

0.57 

6 

13 

0.98 

7 

15 

0.83 

8 

10 

0.57 

9 

6 

0.96 

10 

14 

0.73 

11 

2 

0.70 

12 

3 

0.69 

13 

3 

0.60 

14 

8 

0.87 

15 

9 

0.73 

16 

16 

0.70 

5 

5 

0.53 

14 

7 

0.55 

7 

11 

0.61 

10 

12 

0.58 

Mean  0.71 


Table  2-16 


1 


Agreement  (Ai)  Among  Potentially  Congruent 
Factors  Produced  by  Analysts  ll  and  S 


1 

••  factors  of 

L M 

S 

Al 

Table  2-17 

Agreement  (Ai)  Among  Potentially  Congruent 
Factors  Produced  by  Analysts  K and  S 

Factors  of 


K 

S . 

Ai 

1 

15 

0.84 

2 

4 

0.88 

3 

1 

0.66 

4 

5 

0.61 

5 

1 

0.89 

6 

14 

0.67 

7 

3 

0.77 

8 

16 

0.81 

9 

10 

0.54 

10 

6 

0.87 

11 

8 

0.72 

12 

12 

0.63 

13 

7 

0.69 

14 

9 

0.84 

15 

7 

0.62 

16 

11 

0.51 

11 

2 

0.63 

6 

13 

0.60 

Mean 

0.71 
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Table  2-18 


Average 

Agreement  Among  Four 

Factorial  Structures 

S 

Subject 

Me 

M 

K 

Me 

- 

. 823 

. 760 

. 712 

M 

. 82  3 

- 

. 711 

. 722 

K 

. 760 

. 711 

- 

. 710 

S 

. 712  • 

. 722 

. 710 

_ 

2Ai 

2.29 

2.26 

2.  18 

2.  16 

Cluster  Analysis 


In  an  attempt  to  examine  the  issue  of  factorial  agreement  from  a 
slightlj  different  point  of  view,  a hierarchical  cluster  analysis  was  per- 
formed (Johnson,  1967).  Specificall;-,  the  averaged  agreement  indices 
(Table  2 -18)  were  cluster  analyzed  to  determine  whether  or  not  the  (average) 
agreement  among  the  subjects  formed  more  than  one  cluster.  A single 
cluster  emerged.  A spacial  summary  of  the  analysis  is  presented  as  Ta- 
ble 2 -19.  Again,  the  strength  of  the  agreement  suggesting  a single  cluster  is 
strong.  The  strength  for  cluster  entry  ranged  from  . 71  to  . 82.  This  find- 
ing suggests  support  for  a composite  16  factor  solution. 


Table  2-19 


Cluster  Analysis  Results  Showing  a Single  Structure 
Among  Agreement  Indices  for  Four  Analysts 


Subjects 

Me M S K Strength 


XXX  XXX 

XXX  XXX  XXX 

XXX  XXX  XXX  XXX 


. 82  3 
. 712 
. 710 
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Fifial  Fftctor  An?xlvHis 

The  agreement  results  derived  from  the  data  emerging  from  the 
cosine  analysis  along  with  the  results  of  the  hierarchical  cluster  analysis 
SuPf>ortcd  contentions  favoring  a high  degree  of  agreement  among  the  four 
Snalysts  in  the  perception  of  army  field  messages.  Accordingly,  one  final 
Structure  was  sought  which  be.st  reflected  the  combined  perceptions  of  the 
four  analysts.^  Accordingly,  the  caw  similarity  matrices  of  the  four  anal- 
ysts were  averaged,  correlational  similarities  computed,  and  a single  ma- 
trix was  submitted  to  factor  analysis  and  rotation.  Sixteen  factors  were 
again  extracted.  The  final  rotated  factor  loadings  are  presented  as  Appen- 
dix A to  this  report.  The  eigenvalue  for  each  factor  was; 


Factor 

Eigenvalue 

Factor 

Eigenvalue 

1 

12.  3 

IX 

2.  0 

11 

8.  1 

X 

1.  7 

III 

6.  7 

XI 

1.  5 

IV 

4.  2 

XII 

1.  2 

V 

3.  7 

xin 

1.  1 

VI 

3.0 

XIV 

1.  0 

VII 

2.  5 

XV 

1.  0 

VIII 

2.  4 

XVI 

0.  8 

We  note  that  the  eigenvalue  of  the  last  factor  is  less  than  the  usual 
arbitrary  cut  point  of  one.  Some  degree  of  over  factoring  seems  defensible 
in  the  present  case  where  comprehensiveness  is  sought.  Factors  can  be 
eliminated  if  they  seem  unwarranted  as  the  result  of  use  tests.  However, 
new  empirical  factors  cannot  be  easily  developed. 


Description  of  Fach  Factor 


Each  of  the  16  factors  isolated  in  the  final,  overall  factor  analy- 
sis is  described  in  this  section.  Each  factor  is  also  named  on  the  basis 
of  Stimulus  items  possessing  heavy  loadings  on  the  factor. 


The  stimulus  items  possessing  heavy  loadings  on  factor  I are  pre- 
sented in  Table  2-20.  The  stimulus  items  loading  heavily  on  this  factor  all 
seem  to  be  concerned  with  movement  and  potential  for  movement.  Accord- 
ingly, this  factor  was  named  Movement  and  Capability  for  Movement. 


Table  2-20 


Stimulus  Items  Loading  Heaviest  on  Factor  I-- 
Movement  and  Capability  for  Movement 

Stimulus 

Description 

Loading 

12 

pontoon  sections  support  60  tons 

-.  90 

29  . 

pontoons  are  laid 

-.  84 

17 

speed  is  stressed 

-.  74 

32 

unit  is  mech  unit 

-.  69 

24 

unit  is  amphibious 

-.  67 

57 

...  is  rapid 

-.  63 

5 

troop  unit  moves  SE 

-.  59 

6 

aggressor  vehicles  pass  through 

-.  56 

A number  of  items  concerned  with  numerical  size,  trends,  and 
data  sources  loaded  heavily  on  factor  2.  These  stimulus  items  are  pre- 
sented inTable2-21.  Factor2was  named  Numbers,  Trends,  and  Source 
in  view  of  such  stimulus  items  as:  "small  number  of,  " "decreases,  " and 
"us  radar  section  reports,  " It  is  not  clear  why  sources  fell  on  this  facto 
Possibly,  this  construct  should  be  separated  out  into  a separate  category. 


Tst>I<s  2 -21 


SCfiSulus  ftems  Loading  Heaviest  on  Factor  0 
T^iVmbers.  'I’rends,  and  .Source 

— 

^imulus 

D'esc  ripfion 

Loading 

45 

.•  .•  ••  sMall  namber  of.- .. , 

77 

58 

.•.•.•decreases 

76 

44 

.•  .•  .•  large  volume  of. . . 

• 

o> 

48 

• * 

US  radar  section  reports... 

-.60 

5i 

.•  .•  .•  did  hot  occur 

-.  59 

40 

.•  / . dddurred 

54 

53 

j . is  aerial 

-.  35 

The  third  factor  was  loaded  on  items  describinp  the  character  of 
vafiobs  operations,  e.  g* , "sabotage  team  infiltrates  L'S  lines,  " "aggr, 
FeeGhhaissance  team  seen.  " The  stimulus  items  loading  heaviest  on  this 
factor  are  presented  in  Table  2 =22.  The  factor  was  named  Type  of  Activity. 


Table  2-22 

Stimulus  Items  l.oading  Heaviest  in  I'actor  II1-- 
Tyi'>c  of  Activity 

Stimulus  Descriptinn  Loading 


9 

aggb;  reconnaissance  team  .seen 

88 

25 

patrol  consists  of  20  men 

00 

» 

1 

4 

sabotage  team  infiltrate.s  U.S  lines 

-.81 

io 

fire  is  cxciiangod 

-.77 

33 


Factor  IV  seemed  to  reflect  the  state  of  readiness  of  the  hostile 
forces  as  a function  of  their  training,  motivation/ morale,  and  attitudes. 
This  factor  was  named  Training,  Alorale,  and  Attitudes.  Table  2-23  pre- 
sents the  items  loading  heaviest  on  this  factor. 


Table  2-23 


Stimulus  Items  Loading  Heaviest  on  Factor  IV-- 
Training,  .Morale,  and  Attitudes 


Stimulus 

Description 

Loading 

18 

officers  graduated  war  college 

-.  85 

23 

men  respected  commander 

-.82 

34 

aggr.  units  are  well  trained 

72 

20 

court  martials  are  disclosed 

-.  42 

Factor  V contained  a somewhat  mixed 

set  of  heavily  loaded  stimu- 

lus  items.  On  the  one  hand,  the  factor  alluded  to  communications,  while 
on  the  other  hand,  it  reflected  railroad  oriented  information.  Since  rail- 
road infprmation  would  also  group  with  factor  I (.Movement  and  Capability 
for  Movement),  factor  V was  named  Communications.  The  stimulus  items 
loading  heavily  on  factor  V are  presented  in  Table  2-24. 


Table  2-24 

Stimulus  Items  Loading  Heaviest  on  Factor  V-- 
Communi  cations 

Stimulus Description Loading 

27  aggr.  communication  discipline  poor  ’ -.  87 

15  aggr.  rail  system  is  sabotaged  -.  72 

26  train  is  loaded  with  supplies 


-.  35 


r " ^ 


Factor  VI  was  uniquely  concerned  with  armament  preparation  and 
was  named  accordingly.  The  items  loading  heavily  on  this  factor  are  pre- 
sented in  Table  2-25. 


Table  2-25 

Stimulus  Items  Loading  Heaviest  on 
Armament  Preparation 

Factor  VI-- 

Stimulus 

Description 

Loading 

2 

fortifications  are  prepared 

-.  87 

7 

heavy  gun  emplacements  prepared 

-.  85 

30 

mine  fields  are  prepared 

-.  76 

The  stimulus  items  loading  heavily  on  factor  VII  were  concerned 
with  rifles.  To  generalize  the  factor,  factor  VII  was  named  Small  Arms 
Capability.  The  items  loading  heaviest  on  factor  VII,  Small  Arms  Capabil- 
ity, are  presented  in  Table  2-26. 


Table  2-26 

Stimulus  Items  Loading  Heaviest  on  Factor  VII- - 
Small  Arms  Capability 

Stimulus Description Loading  | 


8 

division  has  new  rifle 

. 93 

11 

range  of  rifle  is  increased 

. 89 

[ 

I 


34 


Any  descriptive  scheme  that  aims  to  classify  military  field  informa- 
ftoh  logically  be  expected  to  contain,  a category  relative  to  the  striking 

pmvet  and  position  of  various  units.  Factor  VIII  was  concerned  with  this  type 
€f  inforttvation.  The  items  loading  heaviest  on  this  factor  are  presented  in  Ta- 
Ble  2-27,  and  the  factor  was  named  Strength  and  Location  of  Units  to  reflect 
tKe  cantent  of  the  heavily  loaded  stimulus  items. 


The  stimulus  items  loading  heaviest  on  factor  IX  are  presented  in  Ta- 
hle  2-28,  The  logical  thread  which  seems  to  weave  these  items  together  is 
Ihtehl  ahd  anticipation.  Accordingly,  the  factor  was  termed  Plans  and  Ex- 
peeiations. 


Table  2-28 


Stimulus  Items  Loading  Heaviest  on  Factor  K-- 
Plans  and  Expectations 

Stimulus 

Description 

Loading 

59 

'commander  states. . , 

-.  76 

55 

enemy  div.  is  alerted  to.  . . 

-.  73 

36 

mission  is  to.  . . 

-.  73 

52 

aggressor  expects  that.  . . 

-.  71 

61 

...  is  preparing  to.  . . 

-.  70 

54 

16CAA  authorizes.  . . 

-.  61 

41 

...  at  this  time 

-.  60 

42 

. . . immediately 

-.  55 

43 

...  in  the  future 

-.  54 

35 

...  is  capable  of.  . . 

-.  50 

Two  factors  seemed  to  be  concerned  with  sources  of  information. 
Evidently,  at  least  in  the  opinion  of  the  analysts  involved,  information  source 
constitutes  meaningful  evaluative  information  to  the  intelligence  analyst.  One 
of  these  factors,  factor  X,  seemed  concerned  with  sources  extraneous  to  the 
field  situation  which  are  possibly  unreliable  (e.  g.  , "civilian  indicates,  " "radio 
broadcasts  indicate").  The  second  of  these  information  source  factors,  factor 
XVI,  was  concerned  with  information  derived  from  written  materials.  Factor  X 
was  named  Extra-Military  Information  Sources,  while  factor  XVI  was  named 
Contents  of  Documents.  The  stimulus  items  loading  heavily  on  each  of  these 
factors  are  presented  respectively  in  Tables  2-2  9 and  2-30. 
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Table  2-29 

Stimulus  Items  Loading  Heaviest  on  Factor 
Extra- Military  Information  Sources 


Stimulus 

Description 

Loading 

50 

civilian  indicates. . . 

. 82 

47 

radio  broadcasts  indicate. . . 

. 72 

Table  2-30 

Stimulus  Items  Loading  Heaviest  on  Factor.  XVI-- 
Contents  of  Documents 

Stimulus 

Description 

Loading 

46 

aggr.  document  indicates.  . . 

-.  75 

16 

document  originates  from  16CAA 

-.  64 

Eactor  XI  loaded  well  on  only  one  stimulus  item--"increases.  " This 
factor  is  essentially  uninterpretable  and  may  be  an  artifact  of  a degree  of  over 
factorization.  Certainly,  the  stimulus  item  "increases"  would  logically  seem 
to  relate  to  factor  H,  Numbers,  Trends,  and  Sources.  In  fact,  the  loading  of 

the  item  "increases”  on  factor  II  was  35.  This  loading  tends  to  support  the 
interpretation  given.  Alternatively,  and  as  discussed  above,  the  "trends  and 
numbers"  aspect  could  be  removed  from  the  definition  of  factor  II  and  treated 
as  a separate  factor. 

Deserter  oriented  information  seemed  to  represent  a unique  considera- 
tion to  the  subjects  involved.  This  type  of  information  was  subsumed  by  factor 
XU,  which  was  named  Deserter  Information.  Information  -relative  to  supplies 
(other  than  armament)  and  equipment  availability  was  embedded  in  factor  XIII, 
named  Logistics.  I’he  stimulus  items  loading  heavily  on  these  two  factors  arc 
presented  in  Tables  2-31  and  2-32. 


Table  2--31 


Stimulus  Items  Loading  Heaviest  on  Factor  XII-- 


Deserter  Information 

Stimulus 

Description 

Loading 

49 

deserter  claims  that.  . . 

. 79 

33 

deserter  entered  friendly  lines 

. 53 

Table  2-32 

• 

Stimulus  Items  Loading  Heaviest  on  Factor  XIII- 

Logistics 

- 

Stimulus 

Description 

Loading 

3 

supplies  are  received 

. 45 

60 

...  is  logistical 

. 41 

The  two  remaining  factors,  XIV  and  XV,  were  weak  but  probably 
meaningful.  Factor  XIV. was  named  Unique  Actions  (e.  g.  , CG  enters  maneu- 
ver area).  This  type  of  information  may  have  factored  separately  because 
the  entrance  of  the  CG  to  an  area  possesses  special  implications.  Such  an 
action  certainly  is  unusual  and  might  warrant  special  consideration  by  the 
intelligence  analyst.  Factor  XV,  the  final  factor,  seemed  concerned  with 
civilian-military  relationships,  as  in  the  stimulus  item  "population  cooper- 
ates with  aggressor"  (loading  . 49),  and  was  named  accordingly-. 
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Discussion  of  Kesul-ts  of  Factorization 


The  message  sample  has  been  manipulated  into  a set  of  factor 
groupings  which  are  logically  coherent  and  show  some  consistency  across 
subjects.  Apparently’,  expert  judges  were  able  to  process  the  informa- 
tion carried  by  each  message  and  converge  on  a similar  structure  for  the 
message  set.  The  resulting  structure  seems  to  possess  construct  validity 
for  development  of  an  organizational  scheme  for  battlefield  linguistically 
oriented  information.  Each  subject  in  the  sample  of  four  persons  permit- 
ted the  production  of  possibly  rival  factor  solutions..  Repeated  challenges 
to  a variety  of  possible  solutions  were  presented  and  systematically  dis- 
counted. . 

As  discussed  relative  to  factor  XI  (page  37)  and  in  view  of  the  eigen- 
values for  the  last  extracted  factor  (page  30).  there  is  some  indication  that 
slight  over  factorization  may  have  been  involved.  In  fact,  some  degree  of 
over,  rather  than  under,  factorization  was  deliberately  superimposed.  It 
is  easier  to  combine  or  eliminate  factors  than  to  develop  new  ones.  If  ex- 
perience indicates  the  need  to  combine  or  eliminate  factors,  such  combina- 
tion/elimination can  be  readily  accomplished.  Moreover,  it  seems  quite 
important  that  an  information  bank  not  lose  information.  An  increase  in 
the  number  of  available  categorization  options  would  seem  to  decrease  the 
potential  for  information  loss. 

We  note  that  the  application  generality  of  the  resultant  factors  is  de- 
pendent on  the  generality  of  the  message  set  employed  as  stimuli.  While 
there  is  no  easy  way  of  know'ing  the  generality  of  the  message  set  employed, 
future  tests  of  the  scheme  are  planned.  These  tests  will  provide  at  least 
initial  insight  into  the  generality  of  the  categorization  scheme  derived. 

There  has  been  at  least  one  prior  study  which  possessed  goals  which 
are  similar  to  those  involved  in  the  present  work  (Miron,  Patten,  & Halpin, 
1975).  Because  of  methodological  differences,  it  is  difficult  to  compare  the 
results  of  the  Miron,  Patten,  and  Halpin  study  with  those  of  the  present  Ap- 
plied Psychological  Services'  study.  In  the  Miron,  Patten,  and  Halpin  study, 
the  approach  dictated  that  the  emergent  structure  be  entirely  dependent  on 
the  adequacy  of  a dictionary  used  to  process  the  messages  involved.  Ac- 
cordingly, the  factor  groupings  of  the  intelligence  reports  in  the  Miron, 
Patten,  and  Halpin  study  were  based  on  overlapping  concept  categories  oc- 
curring among  the  reports.  The  results  of  their  factor  analysis,  consist- 
ing of  five  factors,  were  then  employed  in  an  attempt  to  confirm  the  major 
concept  categories  numbering  some  17. 
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In  the  present  study,  the  resultant  factor  groupings  were  less  con- 
I stricted.  The  factors  grew  out  of  and  are  a direct  product  of  the  messages. 

A categorization  scheme  was  sought  which  is  the  result  of  the  factor  anal- 
ysis, not  the  reverse.  Whereas  the  Miron,  Fatten,  and  Halpin  study  sought 
confirmation  of  major  category  tags  with  the  methodology  of  factor  analysis, 
the  logic  of  the  current  study  required  the  prior  factorial  extraction  as  a 
basis  for  dictionary  devdopmcni.  A comparison  of  the  taxonomies  emerg- 
ing from  the  two  studies  is  presented  in  Table  2-33.  The  present  study 
seems  to  have  isolated  factor^  not  captured  by  the  Miron,  Patten,  and 
Halpin  work.  This  could  be  the  result  of  the  different  approaches  used. 
Additionally,  the  present  study  tended  to  over  factorization  as  a precau- 
tion against  missing  any  vital  categorization.  Provided  that  effective 
-►  cross  referencing  is  employed,  it  seems  that  the  more  fragmented  scheme 

I of  the  present  study  w'ould  provide  less  data  loss  and  more  efficient  infor- 

I mation  insertion  and  retrieval.  Certainly,  many  of  the  factors  of  the  pres- 

|>  ent  study  which  were  not  evidenced  in  the  prior  study  seem  to  possess  mer- 

[*'.  it  on  the  face  of  things,  e.  g.  , communication,  armament  preparations, 

[ * and  training,  morale,  and  attitudes.  More  importantly,  the  present  study 

' includes  all  the  factors  of  the  Miron,  Patten,  and  Halpin  work  plus  other 

factors.  Accordingly,  it  can  be  argued  that  the  results  of  the  present 
study  have  missed  little  of  the  prior  study  and  add  a degree  of  clarifica- 
tion. 


The  work  described  above  suggests  that  the  perceived  structure 
of  the  military  field  messages  analyzed  can  be  represented  by  15(16) 
orthogonal  factors.  The  factors  enlarge  on  and  subsume  the  five  factors 
isolated  by  the  Miron,  Patten,  and  Halpin  work.  Additionally,  it  seems 
that  the  present  factors  are  sufficiently  robust.  Acceptable  agreement 
was  evidenced  among  the  factorial  structures  yielded  by  the  linguistic 
similarity  perceptions  of  the  four  judges  involved.  The  agreement  seems 
to  hold  despite  the  wide  experience  diversity  among  the  judges.  It  is  pos- 
sible that  Army  intelligence  training  tends  to  homogenize  linguistic  per- 
ception. While  the  resultant  factorial  structure  seems  generalizable  to 
friendly  activity,  the  actual  veridicality  of  such  generalization  is  not 
known. 
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Table  2-33 


Comparison  of  Mtron,  Patten,  and  Halpin  Factors  with  Factor  of  Present  Study 


Present  Stu(iy 

Factor  Description 


I Movement  and  capability  of  movement 

II  Number,  trends,  and  source 

III  Type  of  activity 

IV  Training,  morale,  and  attitudes 
V Communications 

VI  Armament  preparation 

VII  Small  arms  capability 

VIII  Strength  and  location  of  units 

IX  Plans  and  expectations 

X Extra  military  information  sources 


Miron  et  al.  Study 

Factor  Description 

I, II  Largo  troop  movements  of  strategic 
importance;  Vehicular  and  small 
scale  trcop  movement 

I, I I Large  troop  movements  of  strategic 
importance;  Vehicular  and  small 
scale  troop  movement 


III  Unusual  small  arms  fire 

I Large  troop  movements  of  strategic 
importance 


Civilian  and  POW  reports  of  low 
reliability 


XI 


Eliminated 


XII  Deserter  information  IV  Deserter  and  POW  reports  of  low 

reliability 

XIII  Logistics  

XIV  Unique  actions  

XV  Civilian  military  relationships  

XVI  Contents  of  documents  
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Factorial  Congruencv:  A Question  of  T.ogic 


The  question  of  degree  of  factor  matching  among  the  four  analysts 
of  the  present  study  may  also  be  examined  by  establishing  a standard  for 
comparison.  Rather  than  compare  factorial  structures  of  the  individual 
analysts  with  each  other  two  at  a time,  each  individual  structure  might 
be  compared  with  the  mean  of  the  group.  Since  each  individual  involved 
in  such  a comparison  is  also  a member  of  the  group,  such  comparisons 
are  statistically  biased  toward  achieving  a good  fit.  Nonetheless,  such 
an  analysis  was  considered  useful  at  least  as  a check  on  the  obtained  re- 
sults relative  to  the  commonality  of  structure  across  the  four  analysts 
who  provided  the  data  reported  and  analyzed  in  the  present  chapter.  To 
achieve  the  standard  structure,  the  raw  similarity  matrices  of  the  four 
analysts  were  averaged,  correlational  similarities  computed  (Stone  et  al.  , 
1970),  and  the  resultant  single  matrix  was  submitted  to  factor  analysis 
and  rotation.  Sixteen  factors  were  extracted.  Each  previously  derived 
individual  factor  solution  was  compared  with  the  average  of  the  group, 
making  a total  of  four  comparisons  of  the  individual  subjects  with  the 
group  mean.  Accordingly,  five  data  sets  were  involved  in  the  four  com- 
parisons. Each  comparison  was  accomplished  in  accordance  with  the 
cosine  rotation  technique  developed  by  Kaiser,  Hunka,  and  Bianchini 
(1971)  and  the  agreement  index  developed  by  Butzkamin  and  Pfeiffer 
(1973).  Since  each  factor  analysis  resulted  in  16  factors,  a 16  x 16 
matrix  of  256  cosines  (representing  the  relationship  of  all  rotated  fac- 
tors of  one  analyst  with  the  group  average)  resulted  for  each  of  the  four 
comparisons.  As  in  the  previous  analyses,  a number  of  controversies 
involving  the  identification  of  congruent  factors  resulted.  The  number 
of  comparisons  involved  in  each  case  ranged  from  17  to  19.  Here,  the 
number  of  comparisons  exceeding  16  roughly  indexes  the  degree  of  con- 
troversy relative  to  goodness  of  fit  achieved.  Tables  2-34  through  2-37 
present  the  resultant  indices  of  agreement.  The  values  of  Tables  2-34 
through  2-37  tend  to  be  higher,  on  the  average,  than  the  previous  individu 

al  comparisons. 

To  obtain  the  average  agreement  of  all  factors  of  each  analysis 
with  all  factors  based  on  the  average  of  the  group,  the  indices  of  agree- 
ment(Ai)\vere  averaged  across  the  total  number  of  potentially  correspond 
ing  factors.  Table  2-33  presents  the  results  of  this  averaging.  The  aver 
age  indices  of  agreement  presented  in  Table  2-38  ranged  from  0.  76  to 
0.  85.  and  are  somewhat  higher  than  the  average  of  the  previous  compari- 
sons’among  individuals.  These  data  suggest  strong  agreement  of  the  in- 
dividual factorial  structures  with  the  group  mean  and  therefore  support 
the  previous  finding  of  strong  agreement  among  the  factorial  struciurcs 
of  data  yielded  bv  the  four  separate  subjects  when  compared  with  cac  t 
other  two  at  a time.  A summary  of  the  matched  factors  is  presented  in 
Table  2-39.  The  table  of  results  suggests  that  with  some  small  ctgtee 
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Table  2-34  . 

Agreement  (Aj ) Among  Potentially  Congruent  Factors  Produced  by  the  A.erage  of 

Group  Versus  Analyst  M'^-  Alone 


AVERAGE 

hf 

1 

7 

0.9272 

2 

. 1 

■ 0.8828 

3 

2 

0.9872 

4 

10 

0.9653 

5 

4 

0.8735 

6 

6 

0.9433 

7 

9 

0.9858 

8 

3 

0.9463 

9 

13 

0.9576 

10 

5 

0.9366 

11 

15 

0.9501 

12 

14 

0.3960 

13 

14 

0.5998 

14 

8 

0.8830 

15 

12 

0.6692 

16 

16 

0.8811 

15 

11 

0.6568 

MEAN 

0.8495 

Table  2-35 

Agreement  (Aj)  Among  Potentially  Congruent  Factors  Produced  by  the  Average  of 

Group  Versus  Ana  I vst  M Alone  


AVERAGE 

M 

Ai 

1 

2 

0.9659 

2 

14 

0.7143 

3 

4 

0.9438 

4 

10 

0.9662 

5 

11 

0.9795 

6 

3 

0.9788 

7 

9 

0.8442 

8 

13 

0.9302 

9 

1 

0.9335 

10 

6 

0.8875 

11 

15 

0.8562 

12 

7 

0.8372 

13 

5 

0.8397 

14 

7 

0.8284 

15 

8 

0.7714 

16 

14 

0.6136 

7 

12 

0.5837 

16 

16 

0.6110 

MEAN 

0.8380 
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Aqreemcnt  (A| ) Among 

Table  2-3C 

Poteritiallv  Cunqruer  t 
Group  Versus;  Analysr 

Factors  Produced  by  the  Avr'ag- 
K Alone 

AVERAGE 

K 

Ai, 

1 

1 

0.8469 

2 

4 

0.9060 

3 

2 

0.9561 

4 

3 

3.8474 

5 

11 

0.9158 

6 

6 

0.8732 

7 

10 

0.9226 

8 

5 

0.6582 

9 

9 

0.6947  ■ 

10 

4 

0.5946 

11 

12 

0.7805 

12 

13 

0.8101 

13 

5 

0.6199 

14 

16 

0.7411 

15 

14 

0.7829 

16 

9 

0.8193 

16 

7 ■ 

0.5535 

4 

8 

0.6283 

12 

15 

0.7516 

MEAN 

0.7736 

Table  2-37 


Agreement  (Aj ) Among 

Potent! a 1 1 y 
Group  Versus 

Congruent 
Analvs+  S 

Factors  Produced  by  the  Ave'ace  of 
A lone 

AVERAGE 

S 

A-i 

1 

15 

0.9646 

2 

5 

0.8265 

3 

4 

0.8939 

4 

1 

0.7183 

5 

2 

0.6951 

6 

13 

0.9649 

7 

6 

0.9676 

8 

1 

0.8652 

9 

3 

0.7895 

10 

16 

0. 5987 

11 

11 

0.6728 

12 

7 

0.9178 

13 

9 

0.7813 

14 

11 

0.8043 

15 

12 

0.8814 

16 

10 

0.3923 

10 

8 

0.5610 

5 

14 

0.657U 

MEAN 

0.7640 
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Table  2-38 


Summary  of  Mean  Agreements  (Aj ) Among  Potentially  Congruent 
Factorial  Structures  Produced  by  the  Average  ot  Aanalysts 
M,  K,  and  S Versus  the  Individuals  Taken  Separately 


COl'fPARISON  . MEAN  A^ 


Group  Average  vs.  0.8495 
Group  Average  vs.  M 0.8380 
Group  Average  vs.  K 0.7738 
Group  Average  vs.  S 0.7640 


Table  2-39 

Summary  of  Matching  Factorial  Structures  Based  on 
the  Averaged  and  Individual  Perceptions  of  Analysts 
MC,  M,  K,  and  S 

Faatovs  by 

GROUP 


AVERAGE 

MO 

M 

K 

S 

1 

7 

2 

1 

15 

2 

1 

14 

4 

5 

3 

2 

4 

2 

4 

4 

10 

10 

3 

© 

5 

4 

11 

11 

2 

6 

6 

3 

6 

13 

7 

9 

9 

10 

6 

8 

3 

13 

5 

1 

9 

13 

1 

© 

3 

10 

5 

6 

0 

16 

11 

15 

15 

12 

11 

12 

© 

7 

13 

7 

13 

14 

15 

d) 

d) 

14 

9 

® 

12 

16 

16 

© 

9 

© 

NOTE:  - Factors  having  questionable  coyigruenay  jvith  the  group  average  have 
circled. 
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of  mismatching,  the  identification  of  corresponding  factors  across  all  anal- 
yses and  factors  can  be  achieved.  Factors  having  questionable  congru- 
ency with  the  group  average  are  circled  in  Table  2-39.  In  all  cases,  the 
factors  having  questionable  congruency  do  not  exceed  three  out  of  the  pos- 
sible total  of  16. 

These*find'ngs  support  the  earlier  conclusion  of  strong  agreement 
based  on  a comple''.e  pairwise  comparison  of  individual  perceptual  struc- 
tures with  each  ot  ier. 
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CHAPTER  III 


TAXONOMY  DEVELOPMENT 


Chapter  I of  the  present  report  described  the  logic,  methods, 
and  results  of  the  steps  completed  to  derive  the  factors  underlying  battle- 
field language.  These  factors,  it  is  held,  can  constitute  a partial  basis 
for  a taxonomy  which  can  structure  and  categorize  battlefield  messages. 

A taxonomy  is  defined  as  "a  classification  of  data  according  to 
their  natural  relationships  or  the  principle  governing  such  classifica- 
tion" (English  English,  1959)  or  as  "a  way  of  simplifying  a compli- 
cated universe  of  individual  events  and  objects  according  to  some  useful 
■way  of  identifying  the  way  in  which  groups  of  individual  (observations) 
have  things  in  common  or  differ  (Miller,  1967). 

Taxonomic  schemes  have  been  found  useful  in  other  contexts. 

The  physical  sciences,  life  sciences,  and  behavioral  sciences  are  three 
areas  in  which  systems  of  organizing  data,  objects,  or  ideas  into  groups 
or  categories  have  been  found  useful.  It  is  possible  that  military  intel- 
ligence information,  w’ith  its  basis  in  the  language  system,  can  profit 
from  a taxonomy  which  would  lead  to  increased  information  understand- 
ing and  interpretive  ease. 

When  considering  such  taxonomies,  it  is  important  to  distinguish 
between  taxonomies  of  attributes  and  taxonomies  of  individuals  (Green  & 
Carmone,  1970).  The  taxonomies  used  to  describe  individual  differences 
in  the  behavioral  sciences  are  usually  classifications  of  attributes  of 
people,  and  less  often  classificationsof  the  people  themselves,  such  as 
sex  and  race.  In  biological  taxonomies,  on  the  other  hand,  the  elements 
are  categories  composed  of  the  biological  organisms  themselves,  such  as 
maple,  pine,  and  oak.  While  such  typological  classifications  of  species  in 
biology  have  survived,  similar  attempts  by  behavioral  and  social  scientists 
to  develop  classification  systems  have  often  lacked  permanency.  This  may 
be  an  artifact  of  the  generalization  that  social  and  behavioral  phenomena 
more  often  fall  between  the  idealized  types  than  fit  them. 

Regardless  of  the  possible  opaqueness  of  a taxonoriiy  for  battle- 
field messages,  we  note  that  such  ta.xonomies  are  held  to  be  useful  by 
others.  In  this  regard.  Sells  (1963)  wrote  in  another  context: 


The  most  obvious  need  is  for  the  development  of  a taxonomy 
and  measurement  technology  of  variables  describing  the 
stimulus  situation ....  But  no  comparable  dimensions  of  the 
stimulus  situation  have  been • systematically  studied,  and 
consequently  only  piecemeal  and  usually  inadequate  account 
can  be  taken  of  variance  attributable  to  the  situation 
(p.5). 

Miller  (1967)  separated  rigorous  and  nonrigorous  taxonomies. 

Miller  contended  that  all  taxonomies  are  nonrigorous,  except  those  which 
are  dependent  on  an  actual  count  or  measurement.  Miller  set  up  the  follow- 
ing objectives  for  any  taxonomy  which  will  be  useful: 

• limited  to  15  to  20  classes 

• classes  can  be  learned  in  a few  hours 

• characterizations  should  foster  anticipa- 
tion of  types  of  errors 

• level  of  detail  should  be  no  greater  than 
possible  prediction  accuracy 

Under  Aliller's  conception,  the  present  taxonomy  would  fall  in  the  "nonrigor- 
ous" category  and  the  present  taxonomy  would  meet  the  first  two  (and  pos- 
sibly the  tliird)  of  Miller's  objectives.  The  fourth  objective  is  not  highly 
relevant  in  the  present  case  since  the  purposes  of  the  taxonomy  are  descrip- 
tive rather  than  predictive. 

The  Taxonomy 

In  order  to  form  the  desired  taxonomy,  the  15  interpretable  factors 
derived  in  Chapter  I and  summarized  in  Table  3-1  were  considered  to  con- 
stitute major  battlefield  message  descriptive  attributes.  However,  since 
the  factors  grew  from  the  linguistically  derived  kernel  and  operator  classes, 
it  seemed  best  not  to  neglect  the  content  analytic  classes  in  which  the  fac- 
tors were  rooted.  Accordingly,  the  second  axis  of  the  taxonomy  was  con- 
sidered to  be  the  classes  from  which  the  factors  were  derived.  Because 
48  classes  seemed  to  represent  excessive  detail  and,  in  some  case,  unde- 
sired partial  redundancy,  the  48  classes  were  reduced  to  27,  These  are 
listed  in  Table  3-2. 


i 
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Table  5-1 


List  of  Factors  in  Taxonomy 

1.  Movement  and  capability  for  movement 

2.  Number,  trends,  and  source 

3.  Type  of  activity 

4.  Training,  morale,  and  attitudes' 

5.  Communication 

6.  Armament  preparation 

7.  Small  arms  capability 

8.  Strength  and  location  of  units 

9.  Plans  and  expectations 

10.  Extra  military  information  sources 

11.  Deserter  information 

12.  Logistics 

13.  Unique  actions 

14.  Contents  of  documents 

15.  Civilian-military  relationships 


Table  3-2 

List  of  Cl  asses  i n Taxonomy 


1.  Composition  of  units 

2.  Markings  and  insignia 

3.  Position  of  forces  and  equipment 

4.  Strength  of  units 

5.  Replacements 

6.  Quantitative  indications 

7.  Weapons  and  equipment 

8.  Analysis 

9.  Characterization  of  missions 

10.  Sabotage 

11.  Actual  and  threatened  contact 

12.  Strategy  and  tactical  doctrines 

13.  Preparations  and  readiness 

14.  Movement 

15.  Chronology 

16.  Information  about  officers 


17.  Competence  of  personnel  and 
equipment 

18.  Mental  and  physical  condition  of 
troops 

19.  Civilian-military  relations 

20.  Sources/methods  of  obtaining  informa 
tion 

21.  Informer  source-military 

22.  Informer  source-nonmiiitary 

23.  High  reliability  source 

24.  Low  reliability  source 

25.  Conditional  phrases 

26.  Unknown  information 

27.  Other 


The  taxonomy  may  be  conceived  as  a matrix  with  the  15  factors 
forming  one  axis  and  the  27  classes  forming  the  second  axis.  Row-col- 
umn intersects  yield  cells  which  are  called  categories.  Any  message 
may  be  assigned  to  one  or  more  categories.  For  example,  the  message 
"infantry  are  trained  in  individual  combat"  would  fall  in  the  category  de- 
fined by  the  intersect  of  the  "Training,  Morale,  and  Attitudes"  factor 
and  the  "Competence  of  Personnel"  class.  The  advantages  of  such  a 
taxonomy  are  lhat  it  could; 


• provided  a basis  for  a battlefield  message  categori- 
zation scheme 


• help  to  assure  complete  categorization  of  any  mes- 
sage 

• help  to  assure  complete  recapture  of  any  message 

• provide  a basis  for  profile  analysis  of  message 
content  to  yield  an  index  of  change  over  time 

• provide  a structured  basis  for  further  research 
into  battlefield  language  and  for  certain  aspects 
of  analyst  training 

• provide  the  basis  for  quantitative  analysis  of  an 
intelligence  data  base  and  of  analyst  decision 
making 

• form  the  substrate  for  at  least  a part  of  a battle- 
field information  management  scheme 

• help  the  battlefield  observer  prepare  clearer, 
improved,  more  comprehensible  messages 

• highlight  the  relationships  among  various  types 
of  message  content 

• improve  communications  accuracy 

• improve  message  content  since  the  messages 
would  be  cast  against  the  backdrop  of  a common 
set  of  factors 

• help  to  standardize  battlefield  language  and  lan- 
guage and  language  interpretation 

• provide  a master  filing  system  of  battlefield  lan- 
guage 
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I The  principal  immediate  application  of  such  a taxonomy,  assum- 

ing required  additional  development  work,  is  a computer  based  support 
system  which  would  facilitate  the  processing  of  intelligence  data.  A 
prior  study  (Miron,  Patten,  & Halpin,  1975)  suggested  the  use  of  com- 
puters in  such  contexts  to  relieve  the  user  of  clerical  activities  and  to 
provide  analytic  and  interpretive  tools  for  transforming  information 
into  intelligence. 

For  such  a system,  both  the  taxonomy  and  the  data  organization 
are  crucial.  Both  will  affect  the  utility  and  meaningfulness  of  the  sys- 
tem to  users.  The  taxonomy  can  provide  a set  of  constructs.  However, 
^ unless  the  constructs  are  presented  in  a structured  and  understandable 

form,  they  may  possess  little  practical  utility. 


Dictionary  Development 

Having  derived  a taxonomy,  the  next  step  appeared  to  be  the  devel- 
opment of  a scheme  which  would  provide  some  basis  for  providing  reliable 
and  accurate  use  of  the  taxonomy.  Such  a scheme  seems  required  in  view 
of  the  richness  of  language  and  the  wide  range  of  words  which  are  used 
almost  interchangeably.  To  this  end,  a "dictionary"  (Appendix  B)  was 
developed.  The  dictionary  was  conceived  as  a multipart  compendium 
which  would  allow  quick,  accurate,  and  reliable  use  of  the  taxonomic 
system. 

"The  dictionary  contains  five  major  sections--factor  definitions, 
class  definitions,  compendium  of  messages,  uniterms,  and  isonyms. 

The  initial  two  sections,  factor  definitions  and  class  definitions, 
contain  definitions  of  the  factors  and  the  classes  included  in  the  taxonomy. 

The  third  section  contains  a compendium  of  messages  which  are 
relevant  to  each  factor/ class  (category).  The  messages  included  are 
the  110  which  formed  the  basis  for  the  factor  analysis.  Each  category 
is  illustrated  by  a set  of  messages.  The  compendium  of  messages  may 
be  helpful  in  several  ways:  (1)  the  messages  included  in  any  category 


help  to  define  that  category,  (2)  a user  with  a new  message  could  match 
the  new  message  to  one  of  the  included  messages;  this  would  help  to  as- 
sure proper  taxonomic  classifications. 


The  fourth  section,  the  index  of  uniterms,  consists  of  a list  of 
alphabetically  arranged  key  terms,  each  followed  by  factor  and  class 
indicators.  The  user  is  thereby  referred  to  the  specific  factors  and 
classes  that  contain  information  relative  to  a specific  key  term. 

If  a taxonomy  user  wished  to  process  very  specific  informa- 
tion such  as  "equipment,  " "sabotage,  " or  "documents,  " reference  to 
these  terms  in  the  Uniterm  section  would  direct  him  to  the  appropriate 
factor(s)/class(es)  containing  the  information.  The  specificity  of  the 
Uniterm  index  allows  a savings  in  time,  since  only  specific  factor(s)/ 
class(es)  may  be  examined  rather  than  an  entire  factor  or  an  entire 
class. 

The  final  dictionary  section,  the  index  of  isonyms  (iso  = same; 
nym  = word),  is  much  like  a thesaurus  providing  semantic  equivalents 
for  various  related  battlefield  words.  The  isonym  list  also  references 
each  listed  word  to  the  taxonomic  category  to  which  the  word  refers. 

The  user  is  accordingly  supplied  with  synonomous  key  terms,  as  they 
appear  in  the  message  compendium  section  of  the  dictionary. 

The  dictionary,  in  its  current  form,  is  experimental,  explora- 
tory, and  exemplary.  Essentially,  the  dictionary  may  be  viewed  as  a 
training  or  job  aid- -offering  information  on  how  to  use  the  taxonomy 
and  where  to  find  information  classified  in  accordance  with  the  taxonomy. 

The  data  base  on  which  the  dictionary,  in  its  current  form,  was 
developed  w'as  small.  If  the  convenience  and  value  of  the  concept  can 
be  demonstrated,  it  can  be  expanded  to  include  a greater  range  of  mes- 
sages and  associated  materials.  It  may  be  that  a taxonomic  classifica- 
tion of  the  messages  relating  to  a given  battlefield  situation  slights  the 
continuous  instability  of  such  situations.  A battle  never  is;  it  is  always 
becoming.  In  addition  to  a wide  variety  of  other  information,  a total  mes 
sage  management  scheme  would  need  to  recognize  the  constant  change  of 
flux  while  accepting  the  utility  of  categorical  analysis. 


CHAPTER  IV 
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Because  the  factors  of  the  taxonomy  were  derived  empirically  and 
because  the  classes  were  derived  from  a structured  linguistic  analysis, 
the  taxonomy  seems  logically  defensible.  However,  the  internal  con- 
structs of  such  a taxonomy  tell  one  little  about  the  pragmatic  v*alue  of  the 
taxonomy.  One  test  of  pragmatic  value,  in  the  present  case,  is  whether 
or  not  intelligence  analysts  can  reliably  use  the  taxonomy.  To  develop 
initial  insights  into  this  question,  two  experiments  were  performed.  In 
the  first  experiment.  Army  personnel  with  experience  in  intelligence  were 
asked  to  read  a set  of  battlefield  scenarios.  They  classified  messages 
relevant  to  the  scenarios  using  a portion  of  the  deri/ed  dictionary  and 
taxonomy.  In  the  second  experiment,  persons  with  similar  experience 
were  asked  to  read  a set  of  scenarios  and  to  state  the  taxonomic  classi- 
fications in  which  they  would  expect  to  find  information  called  for  by  ques- 
tions relevant  to  the  various  scenarios. 


Experiment  1 - -Information  Classification 


The  purpose  of  experiment  1 was  to  test  the  ability  of  experienced 
intelligence  persons  to  classify  messages  in  accordance  with  the  derived 
tsoconomy.  Since  the  total  taxonomy  involves  more'factors  and  classes 
than  could  be  managed  in  the  time  available  for  data  collection,  a sample 
of  factors  and  classes  was  selected  for  inclusion.  The  factors  and  classes 
sampled  were: 

Factors 

A.  Movement  and  Capability  for  Movement 

B.  Training,  Morale,  and  Attitudes 
E.  Communications 

J.  Extra  Military  Information  Sources 

K.  Deserter  Information 

L.  Logistics 

Classes 

3.  Position  of  Forces  and  Equipment 

4.  Strength  of  Units 

6.  Quantitative  Indications 

7.  Weapons  and  Equipment 

10.  Sabotage 

11.  Actual  and  Threatened  Contact 
16.  Information  about  Officers 

18.  Mental  and  Physical  .Condition  of  Troops 

23.  Iligli  Keliabilitv  Source 

24.  Low  Kelialiil ily  Source 
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These  were  -selected  to  represent  a varied  type  of  content  and  a wide 
range  of  potential  message  categorizations.  The  abbreviated  taxonomy 
contained  a possibility  of  60  categorizations  (6  factors  x 10  classes  = 60). 

Methods 


Data  w^ere  collected  in  a group  session  at  the  U.  S.  Army  Intelli- 
gence Center  and  School,  Fort  Huachuca,  Arizona.  After  assembling  the 
group,  a brief  introduction  in  the  purposes  of  the  data  collection  was  giv- 
en. Then  a formal  training  session  in  the  use  of  the  dictionary/taxonomy 
was  presented.  Briefly,  the  instruction  consisted  of  an  explanation  of  the 
tot3.1  taxonomy,  an  explanation  of  the  use  of  the  dictionary,  and  practice 
in  message  classification.  A large  briefing-type  display  of  the  taxonomy 
was  posted  in  front  of  the  room  during  the  training  and  used  as  an  aid. 

The  display  remained  posted  during  the  formal  data  collection.  This  train- 
ing consumed  45  minutes,  and  at  the  end  of  this  time  the  analysts  seemed 
comfortable  with  the  dictionary/ taxonomy. 

After  the  training  session,  stimulus  booklets  were  distributed,  and 
the  analysts  were  asked  to  work  through  the  stimulus  items.  The  analysts 
were  allowed  to  use  the  dictionary  in  any  manner  they  wished  to  assist  them 
in  their  task.  Completion  time  for  all  stimulus  items  and  an  associated 
opinion  questionnaire  was,  on  the  average,.  2 hours  and  15  minutes. 

Stimuli 


The  stimuli  consisted  of  simulated  battlefield  messages  and  associ- 
ated descriptive  scenarios.  The  messages  were  either  taken  verbatim  from 
Army  training  scenarios  or  modeled  after  these  sources.  Two  sample  mes- 
sages are  given  in  Table  4-1. 

Table  4-1 

Sample  Stimulus  Items 

Alternative 

Targeted  Relative  Complexity 

Message  Category*  Categories*  Level 

1.  Deserter  states  his  unit  was  rumored  K-24  - low 

to  be  on  a suicide  mission 

2.  Civilians  indicate  that  14  tank  units  in 
the  1 F div  maneuver  areas  as  of  12  Sep 
are  equipped  with  fording  equipment. 

There  arc  also  amphibious  tanks  mclud 
ed  in  the  force. 


A-7 


A-3,  A-6 
J-3,  J-6, 
J-7 


high 


•Factor  and  class. 


The  set  of  60  messages  was  (Jesigoed  to  meet  two  specificatLoris. 

Bach  message  was  targeted  at  one  of  the  60  factor-class  combinations 
(called  categories)  inclurled.  The  information  contained  in  the  first  sam- 
ple message  in  Table  3-1  can  be  described  by  the  targeted  factor-class 
Pair,  K-24  (where  K stands  for  the  factor  ''deserter  information''  and  24 
stands  for  the  class  "low  reliability  information  source").  Randomly, 
one-fourth.  (15)  of  the  messages  were  chosen  for  use  with  each  of  four  types 
of  scenarios.  *The  messages  and  their  associated  scenarios  were' structured 
to  agree  with  one  another  in  terms  of  time,  day,  date,  location,  and  gen- 
eral type  of  activity.  A second  variable,  complexity  level  of  the  message, 
was  also  varied.  Complexity  refers  to  the  number  of  categories  to  which 
a message  might  pertain;  1 or  2 for  low  complexity;  3,  4,  or  5 for  medium 
complexity;  and  6 or  more  for  high  complexity.  The  first  sample  mes- 
sage in  Table  4-1  was  designed  to  be  relevant  to  only  one  of  the  60  cate- 
gories; The  second  sample  message  was  designed  to  be  relevant  to  six 
categories  (including  the  targeted  category  A-7)  as  indicated  in  Table  4-1. 


^cenajiio_ney^lopment 

The  scenario  associated  with  each  message  provided  a context 
within  which  the  analysts  could  evaluate  the  associated  battlefield  infor- 
fhation.  Each  scenario  was  about  200  words  in  length  and  consisted  of 
a description  of  a current  situation,  what  led  up  to  it,  and  the  types  of 
problems  which  are  currently  on  hand.  One  scenario  was  developed  to 
§kemplify  each  of  the  following  types  of  battlefield  situations;  attack,  re- 
treat, holding  action,  and  logistics.  Since  there  were  60  message  stimuli 
and  four  scenarios,  each  scenario  was  referenced  to  15  messages.  Five 
df  the  15  messages  for  each  scenario  were  randomly  chosen  and  modified 
to  be  of  low  complexity,  another  5 were  randomly  chosen  and  modified  to  be 
of  medium  complexity,  and  the  remaining  5 were  modified  to  be  of  high  com- 
plexity. 

Resnonse  Forms 


T^vo  methods  of  recording  the  categorization(s)  of  the  simulated  bat- 
tlefield messages  were  employed;  (1)  a multiple  choice  approach  wnth  all  60 
messages,  and  (2)  a free  choice  approach  with  a subset  of  these  messages. 
The  distractors  for  the  multiple  choice  aspects  wore  randomly  chosen  from 
those  categories  not  appropriate  to  a given  message.  The  free  choice  meth- 
od allowed  the  analy.sts  to  choose  a limited  number  of  categories  which  they 
considered  to  be  appropriate  to  a given  message.  The  free  choice  response 
form  presented  the  factors  and  classes  in  a matrix  form.  The  analysts  were 
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asked  to  indicate  their  response(s)  by  marking  the  appropriate  cells.  The 
number  of  choices  that  the  analysts  were  asked  to  make  using  this  matrix 
form  was  equal  to  the  number  of  categories  potentially  relevant  to  the  [)ar- 
ticular  message.  Accordingly,  the  analysts  were  advised  of  the  possible 
number  of  category  choices  for  each  message  categorized  using  the  matrix 
answer  form.  The  matrix  response  form  appeared  for  each  item  employing 
this  format  as  follows: 


The  task  of  the  analyst  was  to  state,  W'ithih  a limited  number  of  choices, 
which  factor-class  intersect(s)  (category)  the  presented  message  most  logi 
cally  fell.'’ 


The  free  choice  matrix  form  was  originally  planned  to  be  the  sole 
method  of  recording  categorizations.  However,  pilot  work  indicated  this 
method  to  be  quite  time  consuming.  Therefore,  the  multiple  choice  approach 
described  above  was  developed.  The  matrix  form  allowed  an  assessment,  at 
least  in  part,  of  the  complete  categorization  choices  of  the  analysts.  Also, 
due  to  time  limitations,  analysts  wore  not  asked  to  rate  the  categories  chosen 
from  their  first  choice  (or  most  relevant)  to  their  last  choice  (least  relevant). 
An  example  of  the  multiple  choice  response  form  is  presented  as  Figure  4-1. 
which  shows  a portion  of  the  instruction  page  included  in  each  stimulus  book- 
let. The  parallel  instruction  page  for  the  matrix  response  is  presented  s 
Figure  4-2. 


EXAMPLE:  .Assume  that  you  have  read  a scenario 

about  a battlefield  situation  in  which 
the  friendly  forces  are  engaged  in  a 
holding  action.  Consider  the  following 
message  and  category  choices: 

Message 

SPOTTED  APPROX  40-50  AGO  TROOPS  PROCEEDING 

THROUGH  WOODS  VIC  PA592798  AT  151227. 

Confidence  Score 


Of  the  .categories  given  to  the  right  of  the  message,  category 
A3  (movement  and  capability  for  movement/position  of  forces  and 
equipment)  most  closely  typifies  the  message.  (Verify  for  yourself 
that  the  other  categories  are  not  appropriate.)  Therefore,  you 
would  darken  the  circle  next  to  A3.  The  choice  of  A3  here  seems 
to  be  clearly  correct.  Accordingly,  you  would  select  a confidence 
score  of  "7"  and  enter  your  response  ("7")  in  the  space  provided. 

The  message  with  the  responses  filled  would  therefore  look 
like  this: 


Category  Choices 

ElO  0 D3  O 
A3  O A16  0 
K6  0 


SPOTTED  APPROX  40-50  AGG  TROOPS  PROCEEDING 
THROUGH  WOODS  VIC  PA592798  AT  151227. 


ElO  O D3  0 

A3  « A16  0 

K6  0 


Confidence  Score 


Messages  1-7  refer  to  scenario  I.  Read  scenario  I now.  Then 
categorize  messages  1-7  following  the  instructions  above. 


Figure  4-1,  Partial  example  of  multiple  choice  item  type  response  instructions. 
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Section  2 


Section  2 also  contains  sets  of  messages  and  each  set  is 
again  related  to  a specific  scenario.  To  complcjte  the  questions 
of  each  set  first  road  the  scenario.  Then  for  each  message: 


a)  Select  one  category  which  the  message  seems 
to  fit.  Write  a "1"  in  the  cell  correspond- 
ing to  that  category. 

b)  Select  a second  category  w'hich  the  message 
seems  to  fit.  Write  a "2"  in  the  cell 
corresponding  to  that  category. 

c)  Continue  making  selections  and  write  the 
number  of  each  (3,4,5  and  so  on)  in  the 
appropriate  colls . Limit  your  number  of 
choices  to  the  limit  indicated  above  the 
answer  grid. 

d)  Use  the  dictionary  to  help  you,  if  you 
want  to. 


Example : To  further  clarify  this  procedure  we  will  classify  a 

sample  message.  Assume  that  you  have  read  a scenario  about  a 
battlefield  situation  in  which  the  friendly  forces  are  engaged  in  a 
holding  action.  Consider  the  following  message  and  response  matrix 

Message 

.Spotted  approx  40-50  agg  troops  proceeding 
through  woods  VIC  PA592798  at  151227. 

Limit  your  choice  of  categories  to  3. 


Classes 


Factors 


A B E J K L 


Figure  4-2.  Partial  example  of  matrix  response  item  type  instructions. 
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For  each  scenario  type,  one  low,  one  medium,  and  one  high  complex- 
ity question  w'as  randomly  selected  to  be  answered  using  the  matrix  method. 


\ 

K 


'/* 
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An  additional  dependent  variable,  confidence  in  categorization  choice, 
was  measured  for  various  messages  categorized.  After  each  multiple  choice 
categorization,  the  analysts  recorded  their  confidence  in  the  accuracy  of 
their  categorization  using  a seven  point  scale.  The  categorization  scheme 
employed  was: 


1 = Less  than  5%  confident  that  selected  category  is  correct 

2 = Between  5%  and  20%  confident  that  selected  category  is  correct 

3 = Between  20%  and  40%  confident  that  selected  category  is  correct 

4 = Betw'een  40%  and  60%  confident  that  selected  category  is  correct 

5 = Between  60%  and  80%  confident  that  selected  category  is  correct 

6 = Between  80%  and  95%  confident  that  selected  category  is  correct 

7 = More  than  95%  confident  that  selected  category  is  correct 

Test  Booklets  A and  B 

Each  analyst  received  only  a portion  of  the  60  messages  to  categorize. 
A random  sample  of  30  of  the  multiple  choice  message  categorization  items 
•were  assigned  to  one  booklet  (A).  The  remaining  30  multiple  choice  items 
were  assigned  to  a second  booklet  (B).  All  booklets  contained  the  12  matrix 
form  message  categorization  items. 

Opinion  Questionnaire 

In  addition  to  the  30  multiple  choice  and  12  matrix  response  type  items, 
each  booklet  contained  a final  section  which  sought  to  obtain  the  opinions  of  the 
subjects  relative  to  the  taxonomy.  The  opinion  questionnaire  contained  12 
items; 

1.  an  open  ended  question  relative  to  the  parts  of  the  system 
■which  were  easiest  and  hardest  to  use 

2.  a three  choice  categorical  question  relative  to  .the  amount 
of  detail  included  in  the  factors 

3.  an  open  ended  question  asking  for. examples  of  where  the 
factors  are  too  detailed  or  underdetailed 

4.  a magnitude  estimate  of  the  "naturalness"  of  the  factors 
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5.  a magnitude  estimate  of  the  "understandability"  of  the  factors 

6.  a magnitude  estimate  of  the  "comprehensiveness"  of  the 
factors 

f 7.  an  open  ended  question  asking  for.  any  factors  which  should 

' be  added  or  deleted 

8.  same  as  question  2,  but  relative  to  classes 

^ ' 9.  same  as  question  3,  but  relative  to  classes 

I 

; ♦ 10.  same  as  question  4,  but  relative  to  classes 

11.  same  as  question  5,  but  relative  to  classes 
h 12.  same  as  question  6,  but  relative  to  classes 

r ^ 

K 13.  same  as  question  7,  but  relative  to  classes 

14.  a final  open  ended  question  asking  for  "additional  comments" 


Stimulus  Development 

To  develop  the  necessary  scenarios  and  associated  messages,  a 
former  intelligence  officer  first  developed  the  required  scenarios.  Then, 
the  messages  were  fit  to  the  scenarios.  The  former  intelligence  officer 
then  reviewed  the  materials  for  naturalness  and  reality.  Next,  the  materi- 
als were  tried  on  a small  pretest  sample.  The  results  of  this  pretest  yielded 
areas  of  required  revision  and  indicated  the  need  to  shorten  the  materials  to 
meet  administrative  time  constraints.  This  was  accomplished,  and  the  final 
package  was  again  reviewed  by  the  former  intelligence  officer.  Finally,  the 
booklets  were  produced  in  their  final  form. 


Participants 


The  participants  were  2 9 Army  officers  in  the  initial  stages  of  the 
military  intelligence  staff  officers  advanced  course  at  the  U.  S.  Army  In- 
telligence Center  and  School.  Fort  Huachuca,  Arizona.  Approximate!}- 
90  per  cent  were  Captains  and  the  remaining  10  per  cent  were  First  Lieu- 
tenants, The  mean  time  in  the  Army  of  these  men  was  77.  6 months,  with 
an  average  of.59.  7 months  experience  in  intelligence.  Sixteen  analysts 
completed  stimulus  booklet  A and  13  completed  booklet  B. 

For  analysis,  the  data  of  the  two  groups  were  combined  to  recon- 
stitute the  original  pool  of  60  multiple  choice  items. 

All  analysts  received  those  items  in  which  messages  were  to  be 
classified  into  the  6x10  matrix  representing  the  intersection  of  factors/ 
classes. 


Results 

The  single  most  important  measure  was  considered  to  be  the 
amount  of  agreement  among  analysts  in  selecting  the  alternatives.  This 
was  measured  by  kappa,  an  index  of  agreem.ent  developed  by  Fleiss  (1971). 
Kappa  is  defined  as  the  proportion  of  agreeing  pairs  of  raters  out  of  all 
possible  pairs  of  raters.  This  technique  provides  both  individual  message 
agreement  (P)  as  well  as  overall  agreement  (k),  which  is  simply  the  mean 
of  the  individual  P values. 

According  to  Fleiss,  the  extent  of  agreement  among  the  n raters 
for  the  ith  item  may  be  indexed  by  the  proportion  of  agreeing  pairs  out  of 
all  the  n(n-l)  possible  pairs.  This  proportion  is: 


P.  = Sn;4(n;;-1) 

1 n(n-l)  j. j U 


where:  n 


n(n-l) 


number  of  raters 

number  of  raters  who  assigned  the  ith  item 
to  the  jth  category 

number  of  categories  of  the  nominal  scale 
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'Pho  ovc rail  oxtont  of- ajfi’oomcnt  riiay  tlioa  be  measured  by  the  mean  of  the 
values: 


I 2 \ 

N = total  rtumber  of  items 

^me  degree  of  agreement  is  expected  solely  on  the  basis  of  chance. 
The  eorrection  factor,  as  developed  by  Fleiss  (1971),  such  that  the  overall 
agreement  corrected  for  the  amouftt  of  agreement  expected  by  chance  is 
feappa  (k): 

■p  . "p 

k = — 

1 

e 

^'hefe:  Pg=  mean  proportion  of  agreement  if  the  assignments 
were  made  at  random 

f^ieiss  (l97i)  also  provided  formulas  for  calculating  the  standard 
§ffof  of  kappa;  This  allows  test  of  the  hypothesis  that  k = 0. 

To  illustrate  the  meaning  of  P (proportion  of  agreement  on  an  individu- 
al itiifi)  arid  kappa  (the  mean  of  the  separate  agreements)  an  example  is  pre- 
§6fited  In  Table  4=2; 

N©te  that  the  actual  P values  and  kappa  tend  to  present  a conservative 
^dlht  of  view'.  For  item  2,  in  which  five  of  six  judges  are  in  agreement  P is 
0hly  ; GG  ahd  for  item  3 on  which  four  of  six  judges  agreed  on  one  option  and 
IWo  judges  agreed  on  another  optioh;  the  P value  is  below  . 50.  It  is  evi- 
dfehl  that  P and  k are  tfidre  complex  than  a simple  proportion.  They  are 
proportions  involving  permutations  in  the  numerator  and  denominator  and 
the  total  number  of  possibilities  (the  denominator)  is  sometimes  misleading- 
ly large.  For  example,  for  each  item  there  are  n(n-l)  possible  pairs  of 
agreemehts  (G  raters  taken  2 at  a time  = , = n(n-l)  = 6(5)  = 30  possibil- 

ities. For  item  1 in  Tabled,  there  are  n(n-l)  or  6(5)  - 30  agreements  out  of 
a possible  30  agreements  and  P - 1-.  00.  For  item  2 there  are  n(n-l)  or  (5)(1)- 
20  agreements  out  of  a possible  30  and  P ' . 66.  For  item  3 there  are  n(n-l) 
h'(Vi-l)  or  (4)(3)  + (2)(i)  = 14  agreements  out  of  a total  30  and  P = . 466,  etc. 
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Table  4-2 


Example  of  P and  k Values  Among  Six  Raters  on 
Assigning  Ten  Items  to  One  of  Five  Alternatives 


Item 

Alternative 

% 2 
Sn. . 
1 iJ 

^ 2 
Sn. . - n 
1 iJ 

2 

p _ Sn  -n 
i n(n-l) 

1 

2 

3 

4 

5 

1 

6 

_ 

36 

30 

1.  000 

2 

5 

1 

- 

- 

- 

26 

20  . 

. 666 

3 

4 

2 

- 

- 

20 

14 

.466 

4 

4 

1 

1 

- 

- 

18 

12 

. 400 

5 

3 

3 

- 

- 

- 

18 

12 

. 400 

6 

3 

2 

1 

- 

- 

14 

8 

.266 

7 

3 

1 

1 

1 

- 

12 

6 

.200 

8 

2 

2 

2 

- 

- 

12 

6 

.200 

9 

2 

2 

1 

1 

- 

10 

4 

. 133 

10 

2 

1 

1 

1 

1 

8 

2 

. 066 

P - 

iN 

— SP-  = 

Nj  1 

3.  797 

= .3797 

* — 

10 

« 

• 

k = 

P - P 

e 

. 38 

- . 20 

^ - 225 

1 - P 

1 

- . 20 

7^  ■ 

e 


We  note  that  the  P for  item  3 is  less  than  . 50  in  spite  of  what  seems  to  be  con- 
siderable agreement.  However,  note  that  only  14  pairs  of  agreement  occurred 
out  of  the  possible  30.  The  P values  decrease  rather  rapidly  as  the  amount  of 
agreement  decreases  until  at  item  10,  where  the  worst  agreement  is  illustrated. 
This  item  might  represent  the  actual  possibility  for  pure  random  effects  of  as- 
signment by  the  six  judges.  Consequently,  this  P value  would  be  appropriate  for 
use  as  an  adjustment  for  random  effects  on  agreement  for  each  item. 
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The  obtained  P value  for  each  multiple  choice  item,  by  scenario, 
is  presented  in  Tables  4-3  through  4-6,  along  with  the  corresponding  raw 
agreement  data.  Failure  to  respond  or  missing  data  were  treated  as  lack 
of  agreement.  Accordingly,  the  most  stringent  measures  of  agreement 
was  used.  Agreement  on  each  of  the  items  was  statistically  significant 
with  all  but  three  of  the  60  items  being  significant  at  or  below  the  . 01  lev- 
el. Agreement  for  each  of  the  scenarios  was  also  statistically  significant 
at  the  . 01  level.  The  overall  uncorrected  kappa  value  was  . 738  and  the 
overall  corrected  kappa  value  was  . 664.  These  are  quite  high  and  repre- 
sent considerable  practical  significance  as  well  as  statistical  significance. 


K 


I 


I 

I 

|l 
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Table  4-3 


Number  of  Agreements  and  P Values  Among  Judges  on  Selection  of 
Alternatives  for  Each  Message  of  Scenario  1 


Alternative 


Message 

i 

2 

3 

4 

5 

Pi 

1 

— 

16 

— 

— 

— 

1.000 

2 

1 

— 

15 

— 

■ — 

.875 

3 

— 

— 

— 

15 

1 

.875 

4 

1 

14 

1 

— 

— 

.758 

5 

— 

— 

16 

— 

— 

1.000 

6 

— 

15 

— 

— 

1 

.875 

7 

— 

2 

— 

— 

14 

.767 

8 

6 

1 

— 

1 

5 

.321* 

9 

— 

13 

— 

— 

— 

1.000 

* 

10 

— 

6 

5 

1 

1 

.321* 

11 

— 

11 

— 

— 

2 

.718 

12 

13 

— 

— 

— 

— 

1.000 

13 

— 

— 

— 

13 

— 

1.000 

14 

— 

— 

— 

12 

— 

.846 

15 

11 

1 

1 

— 

— 

.705 

k=  .(W4 

k 1 

corrected  for 

chance=  . 

749 

*Stati 

stically  ; 

R.ignif 

icant  at 

the 

.05  level. 

Values 

above 

.321 

are 

stati 

stically 

significant  at 

or  beloi 

the  .01  level. 


;1 
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Table  4-4 


i 

J 

i- 

I. 


Numbci-  of  Agreements  and  P Values  Among  Judges  on  Selection  of 
Alternatives  for  Each  Messace  of  Scenario  2 


Alternative 

Message  1 2 3 4 5 ^ 


1 1 11  — 

2 — 16  — 

3 — 10  — 

4 — 12  — 

5 3 — w 12 

6 — 16  — 

7 — 16  — 

9 4 5 — 

10  1 12  — 

11  — 3 — 

12  3 1 — 

13  — — 2 

14  — 2 — 

15  2 10  — 


— 4 . .510 

— — 1.000 

4 2 .430 

— 4 . .600 

1 : — .575 

— — 1.000 

— — 1.000 

16  ' — 1.000 

4 — .282* 

— — .846 

8.2  .410 

9 — .500 

1 10  .590 

7 3 .314* 

— 1 .590 


*=  .643 


k corrected  for  chance=  .542 


I 

! 


\ 


’“Statistically  significant  at  .05  level.  Values  above 
.314  are  statistically  significant  at  or  below  the  .01 
level. 
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''‘Statistically  sigrvifica'nt  at 'the  -.05  level.  Values 
^abov;'  -.282  are  statistically  significant  at  or  below 
the  01  level. 


Table  4-6 


r 


Numbel’  St  and  P Values  Among  Judges  on  Selection  of 


An  ftnalyais  of  the  variance  among  the  P values  was  completed  by 
geen&rio.  The  results  indicated  no  statistically  significant  differences 
among  scenarios.  A summary  of  the  results  of  this  analysis  is  presented 
in  Table  4 = 7. 


Table  4-7 


Suttimarv  of  Pictwccn  Scenario  Variance  Analvsis  (P  Values) 


Source 

SS 

df 

MS 

Between  Scenarios 

, 3493 

3 

. 1164 

Within  Scenarios 

3.  2077 

56 

• 

.0573 

Total 

3. 5570 

59 

Confidence 

The  amount  of  indicated  confidence  of  the  subjects  in  their  choices 
Is  presented  by  item  in  Table  4-8.  Translated  into  percentages,  the  over- 
all mean  confidence  value  was  quite  high.  Because  the  confidence  ratings 
were  skewed  tow^ards  the  upper  end  of  the  scale,  median  confidence  ratings 
were  also  calculated.  The  obtained  median  confidence  level  is  presented 
below  by  scenario  along  with  the  corresponding  mean  value: 


Seenario 

Median 

Confidence 

Level 

.% 

Mean 

Confidence 

Level 

% 

I 

Sit  • 

1 

6,  59 

80-05 

6.  09 

80-95 

2 

6.  31 

80-95 

5.  65 

80-95 

i'Mk 

3 

6.05 

80-05 

5,  69 

80-95 

4 

6.  32 

80-05 

6.02 

80-95 

Overall 

6.40 

80-95 

5.  86 

80-95 

t 

The  obtained  individual  .scenario  median  confidence  levels  reflect  consid- 
rral  b 4 iinfidenee  on  the  part  of  the  subjects  in  their  message  categoriza- 


Table  4-8 


Me^  (M)  afKi  Standard  Deviation  (S.  D.  ) of  Subjective  Confidence  by 
MeSvSagc  and  Scenario 


iiessstae 

Scenario  1 

Scenario  2 

Scenario  3 

Scenario 

' 4 

M 

SD 

M 

SD 

M 

SD 

M 

SD 

i 

6.60 

.63 

5.73 

.22 

6.13 

1.13 

6.30 

.88 

2 

6. el 

.36 

6.47 

.64 

4.93 

1.27 

6.53 

.64 

3 

6.40 

.74  . 

4.50 

2.13 

6.73 

.46 

6.31 

.95 

4 

6.40 

.99 

6.40 

.83 

5.46 

1.36 

5.60 

1.40 

5 

6.93 

.26 

5.73 

1.03 

5.53 

1.36 

6.33 

.98 

6 

6.53 

.74 

6.73 

.59 

4.80 

1.42 

6.47 

.52 

7 

6.60 

.63 

6.87 

.35 

5.27 

1.53 

5.93 

1.28 

8 

4.53 

1.56 

6.93 

.26 

6.29 

.99 

6.00 

1.25 

9 

6.13 

.27 

4.57 

2 . 21 

6.36 

.74 

5.21 

2.01 

io 

4.29 

1.68 

6.80 

.58 

5.79 

1.53 

5.21 

1.80 

4i 

5.93 

1.26 

5.86 

1.03 

7.00 

0.00 

6.21 

.80 

42 

5.71 

1.98 

5.29 

1.  38 

6.00 

1.80 

6.07 

.92 

13 

4.71 

2.58 

5.21 

1.72 

4.29 

2.52 

6.36 

.93 

14 

6.57 

.63 

2.79 

2.32 

5.43 

1.28 

5.93 

1.33 

15 

6.S9 

1.86 

4.86 

1.92 

5.43 

1.40 

5.86 

1.51 

dv&bail  iieah 

6.09 

5.65 

5.69 

6.02 

S -.£)•. 

1.28 

1.35 

1.37 

1.22 

I 


i 
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To  determine  if  agreement  was  related  to  confidence,  the  mean 
(Table  4-b)  data  were  plotted  and  the  correlation  between  agreement  and 
confidence  was  calculated.  The  plot  is  presented  as  Figure  4-3.  The  re- 
sulting product  moment  correlation.  . 65,  was  statistically  significant  at 
the  . 01  level  of  confidence.  The  magnitude  of  the  correlation  coefficient 
suggests  that  mean  degree  of  confidence  varies  directly  with  agreement 
on  the  selected  alternative.  The  regression'  equation  relating  agreement 
to  confidence  is  Y'  - 2.2343X+  4.2140,  where  Y'  is  predicted  item  agree 
ment  and  X is  mean  subjective  confidence  for  the  item. 

An  analysis  of  the  variance  of  the  mean  confidence  assignments 
among  the  four  scenarios  was  completed.  -The  results  are  summarized 
in  Table  4-9,  which  indicates  no  statistically  significant  difference  in 
confidence  as  a function  of  scenario  type. 


Table  4-9 

Summary  of  Variance  Analysis  of  Confidence  Ratings 

Source SS df MS 

Between  Scenarios  2.  2251  ! 3 _ 7417 

Within  Scenarios  39.4988  56  . 7053 

Total  41.  7239  59 


_F 

1.  05 
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Matrix  Classification 
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The  matrix  classification  data  were  analyzed  by  extending  Fleiss' 
(1971)  formula  from  each  judge  making  one  assignment  per  item  to  each 
judge  making  more  than  one  categorization  per  item.  The  derivation  of 
this  extension  is  presented  in  Appendix  C.  The  resultant  P values  are 
presented  in  Table  4-10. 


Table  4-10 

P Value  by  Item  for  Matrix  Classification 


« 

Item  Choices  P 

1 3 . 3034 

2 2 . 5787 

3 6 . 4626 

4 1 . 3943 

5 5 .2359 

6 10  .2448 

7 2 .4764 

8 3 .3529 

9 6 .2897 

10  1 ’.3333 

11  3 .1888 

12  6 .4021 

Overall  corrected  for  chance  . 3552 


All  P values  in  Table  4-10  are  statistically  significant  below  the 
. 01  level.  There  was  a considerable  loss  of  agreement  when  the  alterna- 
tives were  increased  from  5 as  in  the  multiple  choice  format  to  60  as  in 
the  matrix  (6  x 10)  format.  The  extent  of  agreement,  however,  was  still 
considerable  when  the  characteristics  of  kappa  are  considered.  . The  rela- 
tionship between  agreement  and  number  of  choices  was  determined.  A 
nonstatistically  significant  correlation,  . 34,  was  indicated.  These  find- 
ings tend  to  imply  that  the  multiple  choice  format  (not  the  number  of 
choices)  tended  to  inflate  the  "agreement"  level. 
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The  matrices  were  further  analyzed  to  determine  whether  factor 
or  class  was  more  important  in  determining  placement  of  the  item.  Fleiss' 
basic  technique  was  modified  and  used  to  determine  the  e.xtent  of  agreement 
for  the  factor  receiving  the  highest  number  of  assignment.^  and  for  the  class 
receiving  the  highest  number  of  assignments.  The  modification  of  Kappa 
was  simply  the  proportion  of  agreeing  pairs  for  a factor  (or  clas.s)  with  the 
most  agreement  for  each  item  over  the  total  po.ssible  number  of  pairs.  This 
is  a measure  of  agreement  md  can  be  interpreted  similarly  to  Kappa.  The 
results  are  pre*sented  in  Table  4-1  1 in  terms  of  the  factor  and  class  receiv- 
ing the  highest  number  of  agreements. 


Table  4-11 


Item 

Agreement  bv 

Factor  and  bv  f'l 

for  Matrix  Categorization 

Choice 

Factor 

Factor 

.Agreement 

Class 

Class 

Agreement 

1 

3 

A 

. 267 

3 

. 132 

2 

2 

A 

. 422 

18 

. 561 

3 

6 

K 

. 337 

3 

.218 

4 

1 

E 

. 393 

7 

. 395 

5 

5 

E 

. 159 

16 

. 073 

6 

10 

A 

. 142 

7 

. 059 

7 

2 

K 

. 424 

18 

. 455 

8 

3 

J 

. 369 

3 

. 246 

9 

6 

J 

. 147 

10 

. 199 

10 

1 

J 

. 318 

10 

. 315 

11 

3 

E 

. 113 

4 

. 089 

12 

6 

J 

. 348 

3 

. 161 

Mean  .2  87 

. 242 

While  Table  4- 1 1 suggests  a slight  advantage  for  the  factor  agreement, 
the  difference  is  not  statistically  significant  when  tested  by  the  sign  test  (two 
tailed).  Moreover,  although  the  variability  appears  much  greater  for  class 
agreement  than  for  factor  agreement,  the  variance  ratio  is  not  statistically 
significant  (F  = 1.  89).  This  suggests  that  the  hypothesis  that  factor  selection 
stability  is  greater  than  class  selection  stability  cannot  be  accepted. 
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Application  of  Theory  of  Signal  Detection  to  Message  Classification 

In  the  first  part  of  this  experiment  (multiple  choice  responses), 
the  analysts  were  asked  to  classify  messages  into  selected  categories 
of  the  developed  taxonomy.  In  this  testing  situation,  the  analyst's  task 
became  that  of  identifying  the  correct  category  from  among  the  altern- 
tives  provided.  One  additional  test  of  pragmatic  value,  in  the  present 
case,  is  whether  or  not  intelligence  analysts  can  effectively  use  and  feel 
comfortable  with  the  taxonomy.  Alternatively,  what  is  the  analyst's 
operating  characteristic?  The  evaluative  construct  used  here  comes 
from  the  theory  of  signal  detection  (Green  &.  Swets,  11)66;  Swets,  1964). 
Since  this  approach  is  well  known,  it  is  only  briefly  considered  in  this 
report.  In  the  signal  detection  literature,  two  types  of  experiment  have 
been  described  (Egan  Clarke,  1966)  to  generate  the  receiver  operating 
characteristic  (ROC).  One  type  of  curve  has  been  termed  the  stimulus 
conditional  ROC  and  the  other  has  been  termed  the  response  conditional 
ROC. 

The  stimulus  conditional  ROC  is  based  on  estimated  probabilities 
P(Y/S)  and  P(Y/N)  conditional  upon  the  occurrence  of  signal  (S)  and  noise 
(N).  Alternatively,  the  yes  (Y)  or  the  no  (X)  response  of  the  subject  is 
conditional  upon  the  events  of  signal  and  noise  that  give  rise  to  the  stimuli. 
According  to  Green  and  Swets  (1966): 

Yes-no  decisions  are  made  by  the  receiver  of  a 
communication  when  he  must  take  one  or  another  • 
action  based  on  his  degree  of  confidence  in  the 
correctness  of  his  identification  response. 

Similarly,  when  the  receiver  sends  back  his 
identification  to  the  source,  the  source  may  be 
required  to  confirm  or  reject  it.  The  analogy 
between  correct  configurations  and  false  alarms, 
has  led  the  analyses  in  terms  of  the  operating 
characteristic.  .^^Ithough  the  analogy  is  not 
exact  in  the  case  of  the  receiver,  based  as  it  is 
on  a discrimination  between  responses  rather  than 
between  stimuli,  the  operating-characteristic 
analysis  serves  its  major  purpose  (p.  313). 
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The  response-conditional  !{OC  is  designed  for  situations  in  which 
the  observer  makes  an  identificatory  response.  This  identification  repre- 
sents the  observer's  best  judgment  as  to  w'hich  event  gave  rise  to  the  in- 
formation received.  To  generate  a response-conditional  HOC’,  the  subject 
must  assign  ratings  of  confidence  to  his  identifications.  The  question  an- 
swered by  this  procedure  is  how  well  the  subject  can  partition  his  ratings 
of  confidence  between  his  correct  and  incorrect  identifications.  Accord- 
ingly, the  response-conditional  HOC  represents  the  relation  between  the 
probability  F(\VC)  that  the  oijserver  will  accept  his  identification  as  being 
correct  w'hen  it  is  in  fact  correct  and  the  proiiability  PiY/1)  that  he  will  ac- 
cept his  identification  as  correct  when  it  is  actually  incorrect.  Previous 
workers  who  have  noticed  the  relevance  of  signal  detection  theory  to  per- 
formance on  mental  tasks  include,  but  are  not  limited  to,  Miles  (I'JTd)  and 
Siegel  and  Pfeiffer  (1969). 

The  binary  techniques  typically  associated  with  signal  detection 
theoretic  analyses  were  not  employed  here  since  more  efficient  use  of 
data  occurs  when  the  rating  method  is  employed.  I'he  relation  b«-tween 
the  binary-decision  procedure  and  the  rating  methoi)  is  that  m the  rating 
method  used  here,  a response  of  "7"  represents  a "yes"  under  a strict 
criterion.  If  the  observation  is  in  fact  correct,  then  PIHy/C')  is  equiva- 
lent to  P(V/C)  where  li-j  is  the  assigned  confidence  rating  "7."  I his  is 
first  computed.  Similarly,  for  the  incorrect  intervals.  1MH-/I)  P(Y/1). 

Next,  the  conditional  probability  ot  a response  of  "6"  to  a correct  choice 
is  computed.  The  following  assumption  is  then  maiie;  observations  ac- 
cepted as  correct  under  a given  criterion  will  also  be  accepted  under  a 
less  strict  criterion.  Therefore,  the  second  value  of  IMY/C)  is  P(H- ♦ C) 

+ P(Hgt  C).  Thus,  the  value  of  P(Y/C  ) cor resjionding  to  the  criterion 
established  by  the  rating  is  the  cumulative  probability; 

1 

P(Y/C)  = S P(IT/C) 
i=7 

The  corresponding  value  of  P(Y/1)  is  found  employing  the  same  logic; 

1 

P(Y/-1)  = V p(R  /I) 
i=  7 ‘ 

Computation  of  P(Y/C)  and  P(Y/1)  continues  over  the  entire  rating  scale 
(i  = 7,  6,  . . . , R)  and  these  data  are  employed  to  generate  the  receiver 
operating  characteristic  (HOC).  A slight  deviation  from  this  described 
procedure  was  necessitated  by  the  lack  of  data  in  the  lower  confidence 
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categories.  Accordingly,  the  data  for  confidence  categories  1,  2,  3 were 
pooled.  This  pooling  reduced  the  number  of  total  confidence  categories 
from  7 to  5 as  follows: 


0.  0 - . 

40 

. 40  - . 60  . 60  - . 80  . 80  - . 95 

.95-1.  00 

(1.  2. 

3) 

(4)  (5)  (6) 

(7) 

(Low> 

^ 

(Confidence  Level) 

— ^-(High) 

The  results  of  the  calculations  for  the  16  analysts  are  presented  in 
Table  4-12.  The  columns  of  Table  4-12  are  headed  by  the  confidence  prob- 
abilities that  define  the  five  rating  categories.  The  first  section,  consisting 
of  the  first  row  of  the  table,  lists  the  frequency  with  which  each  rating  cate- 
gory, Hj,  was  employed.  The  second  section  shows  four  quantities  based  on 
correct  trials,  and  the  third  section  of  Table  4-12  shows  the  corresponding- 
quantities  based  on  incorrect  trials. 

Table  4-12 

Results  of  ROC  Analvsis 


CONFIDENCE 

LEVEL 

0.  0- . 40 

.40-.eo 

. 60-.80 

.80-. 95 

.95-1.00 

(1)  f(Ki) 

16 

33 

63 

118 

250 

(2)  f(C.Ri) 

6 

24 

45 

103 

238 

P(C/Ri) 

.375 

.727 

.714 

.873 

.952 

P(Ri/C) 

.014 

.058 

.108 

.248 

.572 

P(Y/C) 

1.000 

.995 

.928 

.820 

. 572 

(3)  fd.Ri) 

10 

9 

18 

15 

12 

P(I/Ri) 

.62  5 

.273 

.286 

.127 

.048 

P(Ri/I) 

.156 

.141 

.281 

.234 

.188 

P(Y/I) 

1.000 

.844 

.703 

.422 

.188 

NOTE:  - (1)  Section  (2)  is  based  on  413  observations 

(2)  Section  (S)  is  based  on  64  observations 

(3)  N=  480  observations 
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Before  discussing  the  construction  of  the  ROC  curve,  consider  a cal- 
culation which  displays  more  directly  the  intelligence  analysts'  ability  to  use 
the  taxonomy.  Thefirstrow  in  section  2 of  Table  4-12,  labeled  f(C,  Ri),  pre- 
sents the  frequency  of  the  compounded  occurrence  of  correct  trials  and  the 
use  of  category  R^.  The  row  immediately  below  it  lists  estimates  of  the 
conditional  probability  PlC/R^);  specifically,  it  shows  the  proportion  of 
the  observations  placed  in  each. category  that  resulted  from  the  occur- 
rence of  a correct  trial.  For  example,  the  value  0,  952  in  the  extreme 
right  hand  column  is  obtained  by  dividing  the  compound  frequency  (238) 
by  the  total  number  of  observations  placed  in  that  category  (2  50). 

The  quantity  P(C/Rt)  increased  directly  with  increases  in  the  prob- 
abilities that  define  the  confidence  categories.  This  result  is  plotted  in 
Figure  4-4  and  indicates  that  the  analysts  employed  the  rating  categories 
in  a reasonable  manner. 

The  last  two  rows  in  sections  2 and  3 of  Table  4-12  illustrate  the 
calculation  of  the  proportion  of  response  conditional  hits  and  false  alarms. 
The  third  row  in  each  section,  P(Rj/C)  and  P(R^/I),  is  the  reverse  of  the 
conditional  probability  previously  considered.  Thus,  for  example,  the 
value  0.  572  in  the  e.xtreme  right  hand  column  of  section  2 is  obtained  by 
dividing  the  number  of  correct  identifications  made  in  that  category  (238) 
by  the  total  number  of  correct  identifications  (416).  Finally,  the  fourth 
row  in  each  section,  the  response  conditional  hit  and  false  alarm  propor- 
tions corresponding  to  each  of  the  five  decision  criteria,  is  a cumulation 
of  the  quantities  in  the  third  row  moving  from  right  to  left  in  the  table.  It 
represents  the  procedure  of  compressing  rating  data  to  "yes"  - "no"  by 
considering  successively  each  of  the  four  category  boundries  as  defining  a 
"yes"  - "no"  criterion. 

According  to  Coombs,  Dawes,  and  Tversky  (1970),  the  performance 
of  any  sensory  device,  human  or  otherwise,  operating  on  the  basis  of  the 
theory  of  signal  detectability  (TSD)  can  be  completely  described  by  what  is 
called  the  receiver  operating  characteristic  (ROC)  curve.  Figure  4-5  pre- 
sents the  ROC  curve  for  the  16  subjects  of  the  present  stud}'.  Each  subject 
responded  to  30  multiple  choice  items  to  yield  a total  of  480  observations. 
This  value  is  near  the  minimum  of  500  observations  recommended  by  Green 
and  Swets  (1966)  for  such  an  analysis.  The  detection  sensitivity  of  the  anal- 
ysts, d'  = 1.  11,  is  based  on  the  mean  d|  of  the  highest  four  corn'ide-nce  cate- 
gories (7,  6,  5,  4)  presented  in  Table  4-12.  Since  the  indices  of  discrimina- 
bility  (d!y  = 1.  06,  dg  = 1.  12,  dg  = 0.  94,  d^  = 1.  33)  are  nearly  the  same  at 
all  points,  it  seems  that  the  ROC  function  for  the  subjects  and  materials  em- 
ployed is  relatively  bias  free. 


The  question  answered  by  the  ROC  curve  is  how  well  the  intelligence 
analyst  can  partition  his  ratings  of  confidence  between  his  correct  and  incor- 
rect identifications.  Alternatively  stated,  the  question  is  how  sensitive  the 
observer  is  to  the  quality  of  his  own  performance.  It  is  apparent  that  the 
variability  of  the  data  points  about  the  best  fitting  theoretical  curve  is  very 
small. 

Analyst  confidence  was  directly  related  to  correct  message  categori- 
zation and  detection  sensitivity  was  at  a satisfactory  level  when  using  the 
taxonomy.  Accordingly,  the  practical  utility  of  the  taxonomy  is  supported 
from  the  signal  detection  theoretic  point  of  view,  i.  e.,  analysts  are  sensi- 
tive to  the  quality  of  their  own  performance. 

Discussion 

The  results  presented  above  support  the  contention  that  from  the 
point  of  view  of  message  classification,  the  derived  taxonomy  and  associ- 
ated dictionary  possess  considerable  merit.  High  agreement  was  evidenced 
among  users  of  the  system  for  message  classification  purposes.  This 
agreement  seemed  to  sustain  itself  across  scenario  types  and  to  hold  for 
both  the  factors  and  the  classes  of  the  taxonomy.  Moreover,  even  with 
limited  training,  the  subjects  were  confident  about  their  ratings,  and  were 
sensitive  to  the  quality  of  their  performance.  This  confidence  also  sus- 
tained itself  across  scenarios. 

It  is  quite  possible  that  the  obtained  agreement  and  user  sensitivity 
would  attenuate  if  the  total  taxonomy  had  been  employed.  However,  the 
amount  of  this  attenuation  is  a matter  of  conjecture  at  this  junction. 

Accordingly,  from  the  point  of  view  of  message  classification,  the 
derived  system  seems  to  possess  some  merit. 
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Experiment  2 - -Cat  e'^  o r y Query 


Experiment  2 sought  to  investigate  whether  or  not  experienced 
battlefield  intelligence  persons  could  agree,,  with  the  assistance  of  the 
developed  dictionary,  on  which  taxonomic  category  would  most  likely 
contain  the  information  to  allow  answering  questions  relative  to  battle- 
field situations. 

As  for  experiment  1,  only  a portion  of  the  total  taxonon  y was  in- 
cluded. The  factors  and  classes  sampled  in  experiment  2,  were: 

Factors 

C.  Type  of  Activity 

F.  Armament  Preparation 

H.  Strength  and/or  Location  of  Units 

I.  Plans  and  Expectations 

N.  Contents  of  Documents 

Classes 

1.  Composition  of  Units 

5.  Replacements 

8.  Analysis 

11.  Actual  and  Threatened  Contact 
13.  Preparations  and  Readiness 
17.  Competence  of  Personnel  and  Equipment 
19.  Civilian-Military  Relations 


These  were  selected  to  possess  minimum  redundancy  with  the  factors 
and  classes  included  in  experiment  1 and  to  apply  to  a wide  range  of  battle- 
field situations.  In  this  case,  the  abbreviated  taxonomy  consisted  of  35  cate- 
gories (5  factors  x 7 classes  = 35). 

Method 

The  methods  paralleled  those  for  experiment  1.  Data  were  similarly 
collected  at  the  U.  S.  Army  Intelligence  Center  and  School,  Fort  lluachuca, 
Arizona.  The  same  training  syllabus  as  in  experiment  1 was  followed  with 
the  exception  that  the  practice  was  oriented  towards  informatio  i location 
and  retrieval  rather  than  towards  message  classification. 


Stimuli 


i 

I 

t 


The  stimuli  consisted  of  12  scenarios  describing  battlefield  situa- 
tions and  questions  containing  battlefield  conditions  and  activities  relative 
to  these  scenarios.  Two  sample  questions  are  given  in  Table  4-13. 


Table  4-13 

■N 

, Sample  Stimulus  Items;  Experiment  2 

Target 

Question | Category 

1.  Are  the  aggressor  forces  in  the  area  of  H-13 

PA02  87  preparing  for  or  capable  of  a 

i- • major  river  crossing? 

2.  Where  are  the  aggressor  armor  strengh  F-8 

concentrations  in  this  sector? 


Four  sets  of  questions  were  assembled  to  be  relevant  to  5 factors 
and  7 classes.  Each  set  of  35  questions  contained  one  item  targeted  for 
each  of  the  possible  35  factor-class  pairs  (categories).  For  example,  the 
Information  needed  to  answer  sample  question  number  1 in  Table  4-13  is  most 
likely  to  be  found  in  targeted  category  H-13  (factor:  Strength  and/or  Loca- 
tion of  Units,  class:  Preparations  and  Readiness). 


Response  Form 

The  multiple  choice  approach  was  employed  with  all  140  simulated 
battlefield  questions.  The  distractors  for  each  multiple  choice  item  were 
randomly  chosen  from  those  categories  not  appropriate  to  that  question.  A 
sample  page  of  the  response  booklet  is  presented  as  Figure  4-6. 
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Questions  1-6  r.efer  to  scenario  I.  Read  scenario  I now.  Then 
categorize  questions  1-3  following  the  instructions. 


Questions 


Category  Choices 


1. 

What  units  are  engaged  in  fortifying 

Hll 

0 

111 

0 

the  emplacements  on  hill  451  at  the 

C17 

0 

Cl 

0 

present  tiflie? 

FI. 

0 

2. 

Are  there  any  aggressor  engineering  units 

HI 

0 

FI 

0 

in  the  area  or  any  plans  or  preparations  for 

18 

0 

11 

0 

\ 

for  fording  the  Weisse-Elster  river? 

15 

0 

■ ^ 

3. 

Does  the  training  of  replacements  confirm 

N1 

0 

15 

0 

previous  reports  that  the  aggressor  forces 

N17 

0 

11 

0 

are  capable  of  CBR  warfare? 

Nil 

0 

m. 

4. 

Have  sightings  revealed  any  changes  in 

Hll 

0 

F17 

0 

L ' 

position  of  weapons  with  potential 

117 

0 

Fll 

0 

nuclear  capacity? 

H19 

0 

5. 

Are  there  any  indications  that  the  aggressor 

Nil 

0 

111 

0 

aggressor  forces  will  utilize  their 

N17 

0 

117 

0 

1 

nuclear  weapons  against  friendly  units? 

11 

0 

f 

6 . 

What  are  the  general  plans  of  the  ■ 

FI 

0 

H5 

0 

t 

9,ggressor  during  the  standdown  period? 

F8 

0 

18 

0 

i 

F5 

0 

Questions  7-12  refer  to  scenario  II.  Read  scenario  II  now. 
Then  categorize  questions  7-12  following  the  instructions. 


7.  Send  G2  information  on  sightings  of 
any  weapons  with  nuclear  capability? 


117  O 
H17  O 
H19  O 


H13  0 
C17  0 


Have  the  local  civilians  become  aware  of  C13  O C19  0 
aggressor  preparation  for  possible  use  of  F19  O 15  0 
nuclear  weapons?  F17  O 


Have  any  local  farmers  been  recruited 
for  guerrilla  units? 


119  0 
F13  0 
H13  0 


F5  0 
N8  0 


Do  documents  found  in  hidden  guerrilla 
supply  cache  (VIC  PA432472)  indicate  the 
plans  of  the  irregulars  in  this  area? 


N13  0 
II  0 


Figure  4-6.  Page  of  experiment  2 response  booklet. 
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Test  RookUrts  A And  R 


Each  analyst  received  only  a portion  of  the  140  questions  to  cate- 
gorize. A random  half  of  the  questions  for  each  scenario  was  assigned  to 
one  booklet  (A).  The  remaining  questions  relevant  to  that  scenario  were 
assigned  to  a second  booklet  (B).  The  two  forms  of  the  test  were  organ- 
ized into  two  versions  of  test  booklets  to  facilitate  the  data  collection.  Fif- 
teen analysts  completed  booklet  A and  16  completed  booklet  B. 


Feedback  Questionnaire 

The  same  opinion  questionnaire  as  included  in  experiment  1 was 
Included  in  experiment  2.  ' 


Scenario  Development 

Scenarios  similar  to  those  used  in  experiment  1 were  developed 
to  provide  a context  for  the  information  processing  task.  Twelve  sce- 
narios, three  scenarios  of  each  of  the  four  types  used  in  experiment  1, 
were  developed.  Each  scenario  type  was  queried  relative  to  all  35  tar- 
geted categories.  The  35  targeted  categories  were  randc  ’ ' divided 
among  the  three  scenarios  of  a type  (with  the  constrain  he  distribu- 

tion of  number  of  questions  per  scenario  be:  12.  12,  ai. 

Stimulus  Development 

Stimulus  development  paralleled  that  for  experiment  1,  including 
initial  preparation  by  the  same  former  intelligence  officer,  pretest,  revi- 
sion, and  final  assembly  in  the  form  of  two  booklets.  Completion  of  each 
stimulus  booklet,  including  the  associated  opinion  questionnaire,  took 
about  one  hour  and  45  minutes. 

Participants 

The  participants  in  experiment  2 were  30  captains  and  one  first 
lieutenant  enrolled  in  the  military  intelligence  staff  officers  advanced 
course  at  h’ort  Iluachuca.  These  officers  had  been  in  the  Army  for  slight- 
ly more  than  6 years,  on  the  average.  The  range  was  4 years  to  slightly 
over  12  years.  On  the  average,  the  analysts. had  4 years.  P months  ex- 
perience in  intelligence  work.  Two  of  the  analysts  had  no  intelligence 
experience,  while  the  analyst  with  the  greatest  amount  of  military  intel- 
ligence experience  liad  11  years  and  8 months  of  such  duty. 
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Results 


The  kappa  statistic  (Fleiss,  1971)  was  calculated  to  determine  the 
extent  of  agreement  among  the  analysts  on  the  task  of  identifying  that  cate- 
gory which  should  be  queried  for  obtaining  information  needed  to  answer  a 
question  relative  to  the  scenario  situation.  The  analyses  were  performed 
separately  on  Forms  A and  B.  In  addition,  a kappa  on  the  combined  data 
(Form  A plus  Form  B)  was  also  obtained.  The  obtained  values,  corrected 
for  chance  kappa,  were:  Form  A = .52;  Form  B = .43;  overall  = . 48. 

On  the  basis  of  chance  alone,  the  following  proportions  of  agree- 
ment would  be  expected:  Form  A = .21;  Form  B = .20;  overall  = . 20. 

Standard  errors  of  kappa  were  obtained  to  examine  the  hypothesis 
of  no  agreement  beyond  chance.  The  obtained  standard  errors  of  kappa 
were:  Form  A = . 006;  Form  B = . 005;  overall  = . 005.  Calculation  of  the 
statistical  significance  of  the  obtained  kappa  values  from  these  standard 
error  values  indicates  that  the  obtained  kappa  values  are  significantly  dif- 
ferent from  chance. 

Kappa  values  were  also  calculated  for  each  item  by  scenario.  These 
statistics  are  presented  in  Table  4-14,  which  also  presents  the  raw  agree- 
ment data  and  the  P values  by  item.  Each  of  the  12  scenario  kappa  values 
presented  in  Table  4-14  is  statistically  significant  at  or  below  the  . 01  level. 

Examination  of  the  statistical  significance  of  the  agreement  findings 
on  individual  items  indicated  that  agreement  on  1 7 items,  those  whose  P 
value  was  less  than  . 295,  did  not  reach  significance.  The  agreement  on 
Iwo  items  was  significant  at  the  . 05  level;  these  were  items  whose  P values 
were  between  .295  and  . 313.  The  agreement  among  the  analysts  on  the  re- 
maining 121  items  was  statistically  significant  at  or  below  the  . 01  level. 


Table  4-14 


Number  of  Agreements  and  P Values  Among  Judges  on  Selection  of 
Alternatives  for  Eaeh  Item  and  Scenario 


Alternative 


Item 

1 

2 

3 

5 

4 

Pi* 

1 



14 

1 

.867 

2 

— 

15 

— 

— 

— 

.524 

3 

10 

1 

1 

— 

3 

.457 

4 

— 

— 

. 1 

2 

12 

.638 

l-l 

5 

5 

1 

— 

9 

— 

.438 

0 

6 

3 

— 

10 

1 

2 

.408 

•H 

f. 

7 

3 

4 

— 

— 

9 

.375 

H 

ci 

8 

— 

5 

— 

9 

2 

.392 

c 

0> 

9 

6 

2 

— 

3 

5 

.242 

c; 

w 

10 

— 

— 

2 

12 

2 

.567 

11 

1 

— 

— 

— 

15 

.875 

.395 

k 

corrected  for 

chance= 

.526 

1 

2 

- 

12 

1 

.638 

2 

1 

12 

— 

2 

— 

.638 

3 

— 

5 

3 

1 

6 

.267 

4 

— 

5 

6 

— 

4 

.295 

5 

3 

— 

1 

— 

11 

.552 

0 

6 

1 

8 

1 

4 

1 

.324 

*H 

u 

7 

— 

5 

— 

9 

2 

.392 

OS 

8 

— 

9 

— 

7 

— 

.475 

c 

0) 

9 

12 

— 

3 

— 

1 

.575 

o 

CQ 

10 

— 

1 

3 

12 

— 

.575 

11 

1 

13 

— 

1 

1 

.650 

12 

— 

1 

4 

11 

— 

.508 

*= 

.324 

k 

corrected  for 

chance= 

.491 

k=  .432 


k corrected  for  chance=  . 551 


1 

10 

— 

3 

2 

— 

.467 

2 

14 

— 

— 

— 

1 

.867 

3 

8 



7 

— 

— 

.467 

4 

3 

— 

12 

— 

— 

.657 

5 

— 

— 

— 

1 

14 

.867 

6 

— 

15 

— 

— 

— 

1.000 

7 





2 



13 

.654 

8 

4 



2 



9 

.354 

9 

6 

3 

6 



— 

-271 

10 



1 

4 

5 

4 

.175 

11 

8 

1 

— 

— 

6 

.354 

12 

— 

7 

1 

7 

— 

.346 

Jc=  401 


k corrected  for  chance=  .539 


Scenario  VIII  Scenario  VII. 


Table  4-14  (cont.  ) 


Alternative 


Item 

1 

2 

3 

4 

5 

Pi* 

1 

1 

1 

13 

.743 

2 

4 

11 

— 

r 

— 

.581 

3 

1 



1 

13 



. .743 

4 

6 

4 

3 

2 

— 

.238 

5 

2 

10 

2 

— 

1 

.448 

6 

16 

— 

— 

— 

— 

1.000 

7 

12 

— 

— 

3 

1 

.575 

8 

— 

11 

2 

1 

■ 2 

.475 

9 

4 

— 

— 

2 

10 

.433 

10 

' 

— 

3 



13 

.675 

11 

10 

1 

1 

4 

— 

.425 

k= 

.445 

k 

corrected  for 

chance  = 

1 

1 

1 

12 



1 

.629 

2 

4 



5 

— 

6 

.295 

3 

1 

— 

* — 

12 

2 

.638 

4 

— 

14 



1 



.867 

5 

1 



10 

— 

4 

.486 

6 

6 

1 

4 

— 

4 

.257 

7 

— 

1 



8 

7 

.408 

8 

11 

2 

1 

1 

1 

.467 

9 

1 

6 

— 

6 

3 

.275 

10 

15 



1 

— 

— 

.875 

11 



1 

. 15 





.875 

12 

5 

1 

3 

— 

7 

.283 

*= 

.402 

k 

corrected 

for 

chance= 

i 

i 


k 1 
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Scenario  X Scenario  IX 


Table  4-14  (cont.  ) 


Alternative 


1 

4 

- — 

3 

8 

■ — 

.352 

2 

— 

— 

14 

1 

■ — 

.867 

3. 

2 

10 

— - 

3 

— 

.467 

4 

— 

— 

13 

1 

1 

.743 

5 

— 

— 

— 

3 

11 

.548 

6 

— 

3 

6 

— 

6 

.314 

7 

1 

8 

— 

1 

6 

.35a 

8 

2 

5 

— 

— 

9 

.392 

9 

16 

— 

— 

— 

— 

1.000 

10 

14 

1 

— 

— 

1 

.758 

11 

1 

1 

3 

6 

5 

.233 

12 

1 

1 

— 

14 

.758 

*= 

k 

corrected  for 

chance= 

1 

2 



12 



1 

.638 

2 

— 



1 

11 

3 

.552 

3 

— 

1 

— 

1 

13 

.743 

4 

— 

— 

15 

— 

— 

1.000 

5 

— 

3 

2 

7 

3 

.267 

6 





15 

- 

1.000 

7 

— . 

— 

14 

1 

1 

.758 

8 

1 

— 

3 

3 

8 

.279 

9 

— 

— 

2 

— 

14 

.767 

10 

2 

3 

1 

10 

— 

.408 

11 

2 

11 

— 

3 

— 

.492 

k= 

.4S9 

k 

corrected 

t for 

chance- 

Scenario  XII 


Table  4-14  (cont.  ) 


Alternative 


1 

2 

3 

4 

5 

Pi* 

1 







15 

1.000 

2 

3 

— 

12 

— 

— 

.657 

3 

— 

— 

— 

— 

15 

1.000 

4 

— 

— 

3 

12 

— 

.657 

5 

— 

7 

8 

. — 

— 

.467 

0 

6 

1 

— 

— 

3 

11  . 

.552 

•H 

7 

1 

13 

— 

1 

1 

.650 

a 

8 

9 

5 

— 

1 

. 1 

.383 

mm 

0) 

9 

— 

1 

— 

5 

10 

.458 

o 

CO 

10 

2 

14 

— 

— 

— 

.767 

11 

1 



13 

1 

.646 

12 

— 

14 

1 

— 

1 

.758 

.563 

k 

corrected 

' for 

chance= 

666 


1 4 — 

2 3 1 11 

3 — 8 2 

4 1 10  2 

5 1 14  — 

6 — 11  3 

7 — 16  — 

8 — 1 — 

10  2 3 8 

11  11  — 

12  1 15  _ 


3 


3 2 

1 — 


1 — 

15  — 

8 8 

2 — 
2 12 


.352 

.552 

.314 

.433 

.867 

.552 

1.000 

.875 

.467 

.271 

.556 

.875 


.435 


k corrected  for  chance^  .593 


*<.295;  not.  statistically  significant 
>•295  <.313;  statistically  significant  between  .05  and 
.01  levels  of  confidence 

>.313;  statistically  significant  at  or  below  .01  level 
of  confidence 
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An  analysis  of  the  variance  of  the  P values  across  scenarios  iras 
completed.  A summary  of  the  results  of  tliis  variance  analysis  is  presented 
inTable4-15.  The  results  indicate  no  statistically  significant  differences 
among  the  P values  of  the  various  scenarios. 


Table  4-15 


Summary 

of  Between  Scenario  Variance  Anal', 

nsis  (P  Values) 

F 

Source 

SS 

df 

MS 

Between  Scenarios 

. 7544 

11 

. 0686 

1.  25 

Within  Scenarios 

• 6.  9727 

127 

. 0549 

Total 

7.7271 

128 

Accordingly,  the  experiment  2 analysts  demonstrated  statistically 
significant  agreement,  and  the  extent  of  agreement  did  not  vary  across  the 
12  scenarios. 


Agreement  Between  Experiments 

The  chi  square  technique  was  used  to  test  the  significance  of  difference 
between  the  kappa  values  of  experiment  1 and  experiment  2.  Specifically,  the 
question  of  concern  was:  does  the  agreement  among  the  analysts  in  experiment  1 
(whose  task  was  to  classify  messages  into  the  categorization  scheme  employed) 
differ  significantly  from  the  agreement  found  among  the  analysts  in  the  second 
experiment  (whose  task  was  to  identify  categories  they  would  select  to  obtain 
information  required  to  answer  intelligence  related  questions)?  The  overall 
kappas  (across  both  forms  in  each  experiment)  were  used  in  this  analysis. 

The  resulting  chi  square,  3.24,  was  not  statistically  significant.  This  indi- 
cates that  although  agreement  tended  to  be  higher  among  the  analysts  in  the 
first  experiment  as  compared  with  the  analysts  of  the  second  experiment,  the 
difference  was  not  large  enough  to  indicate  that  the  two  task  types  are  independ- 
ent in  the  population  considered. 
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Responso  Correctness 


The  previous  analj'ses  pertained  to  the  agreement  among  the  Army 
intelligence  personnel  sampled  on  the  identification  of  a category  they  would 
be  likely  to  query  if  they  were  searching  for  specific  intelligence  data.  To  de- 
termine insight  into  response  accuracy,  the- responses  to  the  combined  70 
questions  were  scored  on  an  absolute  scale  for  correctness  of  response. 

The  scoring  key  was  based  on  the  responses  of  the  research  team,  staff 
members  of  Applied  Psychological  Services.  The  mean  numbei  of  correct 
responses  (across  both  forms)  was  49  or  70  per  cent,  with  a standard  deviation 
of  6.46.  The  range  of  correct  responses  was  from  3 1 to  6 1 (44  to  87  per  cent). 

Kendall's  coefficient  of  concordance  was  applied  to  the  accuracy 
scores  of  the  analysts  of  experiment  2 as  they  occurred  on  each  of  the  12 
scenarios.  The  coefficient,  calculated  in  this  manner,  provides  an  initial 
index  of  systematic  variation  in  accuracy  as  a function  of  scenario.  The  re- 
sultant coefficient  of  concordance  (corrected  for  tied  ranks)  was  . 106.  This 
is  interpreted  as  indicating  minimum  correlation,  if  any,  between  scenarios 
and  accuracy. 

To  verify  the  prior  interpretation,  an  analysis  of  variance  was  per- 
formed on  the  accuracy  data  to  determine  if  a statistically  significant  differ- 
ence existed  among  the  accuracy  scores  for  the  scenarios.  The  results  of 
the  analysis  are  summarized  in  Table  4-16. 


Table  4-16 

Summary  of  Analysis  of  Variance  for  Scenarios  (Accuracy  Scores) 


Source  SS  df  MS  F 


Between  Scenarios 

12,  869.  26 

11 

1,  169.  93 

2.  83- 

Within  Scenarios 

148,  867.  29 

360 

413. 52 

Total 

161,  736.  54 

371 

* Statistically  significant  at  the  . 01  level. 


The  F value,  2.  83,  was  statistically  significant  at  the  . 01  level.  Tukey's 
gap  test  was  applied  to  the  data  to  determine  those  scenarios  which  were 
significantly  different  (Table  4-17). 
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Table  4- IV 


Gaps  Between  Scenario  Means 
MEAN 


SCENARIO 

MEAN 

62 

63 

66 

68 

69 

70 

71 

73 

74 

6 

86 

© 

© 

(© 

■© 

© 

© 

© 

© 

11 

74 

© 

11 

8 

6 

5. 

4 

3 

1 

3 

73 

11 

10 

7 

5 

4 

3 

2 

9 

71 

9 

8 

5 

3 

2 

1 

j 

8 

• 70 

8 

7 

4 

2 

1 

4 

. 69 

7 

6 

3 

1 

i 

12 

69 

7 

6 

3 

1 ■ 

1 

68 

6 

5 

2 

i 

7 

68 

6 

5 

2 

f' 

f 

10. 

66 

4 

3 

5 

63 

2 

\ 

2 

62 

Following  Tukey's  procedure,  a D of  11.  55  (significant  at  the  . 05  level) 
was  obtained.  A comparison  of  the  differences  of  the  scenario  means 
with  D indicated  that  the  scenario  6 scores  were  significantly  different 
from  the  other  scenarios.  Scenario  6 possessed  the  highest  mean  accu- 
racy score.  A second  gap  was  found  to  exist  between  scenario  11  and 
scenario  2.  No  other  statistically  significant  gaps  were  isolated.  In 
view  of  the  large  number  of  possible  gaps  when  12  mean  values  are  in- 
volved, the  finding  of  only  a limited  number  of  gaps  is  interpreted  as 
indicating  that  scenario  type  did  not  materially  affect  accuracy.  Examina- 
tion of  the  individual  scenarios  indicated  no  obvious  differences  between 
scenario  6 and  the  other  scenarios  or  between  scenarios  11  and  2. 

These  accuracy  results  are  promising  from  the  point  of  view  of 
the  ease  of  handling  the  categorization  system  employed  in  the  study. 

Still  better  and  more  uniform  performance  might  be  anticipated  with  ad- 
ditional experience.  In  view  of  the  minimal  training  involved,  the  anal- 
ysts were  able  to  perform  with  moderate  accuracy,  were  again  self  con- 
sistent among  themselves,  and  querying  accuracy  did  not  seem  to  be  seri- 
ously debilitated  by  scenario  type. 

Discussion 

The  results  of  experiment  2 tend  to  be  in  conformity  with  those  of 
experiment  1.  Again,  substantial  agreement  was  evidenced  among  anal- 
ysts and  the  agreement  was  scenario  independent.  Additionally,  adequate 
accuracy  was  demonstrated  in  experiment  2.  and  the  accuracy  was  largely 
scenario  independent.  When  the  experiment  2 results  are  coupled  with 
the  results  of  experiment  1.  it  seems  that  the  derived  system  can  be  con- 
sidered to  be  self  contained  both  from  the  points  of  view  of  message  clas- 
sification and  category  inquiry.  The  coupled  results  support  contentions 
favoring  the  possible  potential  of  the  derived  system. 


( 
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We  note  that  although  the  difference  between  the  overall  kappa  val- 
ues for  the  two  experiments  was  not  statistically  significant,  the  trend 
was  towards  higher  agreement  in  experiment  1 than  in  experiment  2.  The 
reason(s)  for  this  trend  are  not  entirely  obvious.  The  inquiry  task  seems 
more  opaque  than  the  classification  task,  and  this  might  suggest  some 
need  for  additional  system  clarification  relative  to  inquiry.  It  is  also  pos- 
sible that  qualicative  differences  existed  in  the  training  for  the  two  groups. 
While  the  same  training  syllabus  was  followed  for  both  groups,  different 
administrators  were  involved.  This  was  necessitated  because  the  schedul- 
ing of  the  participants  required  that  ooth  data  acquisition  sessions  take 
place  at  the  same  time.  While  the  two  administrators  "rehearsed"  togeth- 
er and  attempted  to  "standardize"  as  much  as  possible,  the  conjecture  of 
across  administrator  differences  cannot  be  rejected.  Additionally,  the 
possibility  of  group  differences  remains  open.  However,  this  possibility 
does  not  seem  tenable.  The  subjects  were  assigned  by  intelligence  school 
personnel  to  the  two  groups  in  such  a manner  which  seems  to  assure  ran- 
dom assignment. 

Other  explanations  for  the  findings  relative  to  the  trend  implying 
differences  between  the  inquiry  and  the  retrieval  tasks  remain  open.  The 
"tests"  for  each  type  of  task  were  not  equated  for  difficulty,  different  sce- 
narios were  involved,  and  a larger  number  of  scenarios  was  involved  in  ex- 
periment 2.  For  example,  experiment  1 was  based  on  four  scenarios  rep- 
resenting each  of  the  following  types  of  activity:  retreat,  attack,  logistics, 
and  holding.  Three  of  each  of  these  types  of  scenario  were  involved  in  ex- 
periment 2.  Additionally,  the  categorization  scheme  was  drawn  directly 
from  messages--not  from  questions.  Accordingly,  one  might  anticipate 
greater  ease  for  direct  categorization  of  messages  than  for  questions  rela- 
tive to  messages.  Finally,  it  is  possible  that  the  two  types  of  tasks  rest 
on  different  mental  operations  or  on  different  levels  of  the  same  mental  oper- 
ation. It  seems  that  the  data  bank  query  task  of  experiment  2 demands  more 
data  evaluation  than  does  the  input  task  of  experiment  1.  Moreover,  an  anal- 
yst may  possibly  read  more  implications  into  a question  than  into  a concrete 
message,  and  he  may  perceive  more  potential  relationships. 

From  the  point  of  view  of  the  experimental  results,  it  seems  that 
a viable  system  has  at  least  been  partially  achieved.  However,  as  with 
any  evolving  system,  additional  clarification  may  be  required  for  maxi- 
mum utility.  We  note  that  the  methods  of  study  and  analysis,  here  em- 
ployed, were  largely  unique.  There  were  no  models  to  rely  on  and  study, 
and  analysis  methods  were  developed  in  parallel  with  the  taxonomic  sys- 
tem. The  methods  developed  here  can  form  at  least  a partial  foundation 
for  future  work  of  the  sort  which  was  the  concern  of  the  two  experiments. 


Opinion  Questionnaire  Hesults 


After  the  completion  of  the  data  acquisition  forms  for  each  experi- 
ment, the  analysts  completed  an  opinion  questionnaire  consisting  of  14 
items  on  various  aspects  of  the  classification  system.  Some  of  the  items 
were  open  ended,  some  were  multiple  choice,  and  others  called  for  mag- 
nitude estimation. 


Factors 

When  asked  whether  the  factors:  (a)  were  overly  detailed,  (b)  had 
the  proper  amount  of  detail,  and  (c)  were  not  detailed  enough,  the  majori- 
ty (59  per  cent)  of  the  analysts  in  experiment  1 (whose  task  involved  the 
categorization  of  intelligence  messages)  indicated  that  the  factors  had  the 
"proper  amount  of  detail.  " A fair  proportion  of  the  sample,  33  per  cent, 
thought  the  factors  should  be  "more  detailed.  " Only  two  analysts  (8  per 
cent)  indicated  that  they  considered  the  factors  to  be  "overly  detailed.  " 

The  analysts  of  experiment  2,  on  the  other  hand,  were  almost  equally 
divided  ov'er  the  three  options.  These  results  suggest  that  for  the  task 
of  categorizing  messages,  the  factors  are  sufficiently  detailed,  but  that 
additional  factors  might  be  helpful  for  the  inquiry  task.  We  note  that  the 
quantitative  kappa  values  indicated  less  agreement  among  the  analysts 
who  performed  the  experiment  2 inquiry  task  as  compared  with  those  who 
performed  the  experiment  1 message  classification  task.  This  higher 
state  of  disagreement  may  be  mirrored  in  the  opinions  of  the  experiment  2 
analysts  relative  to  the  amount  of  detail  included  in  the  present  system. 

Very  few  of  the  analysts  of  either  experiment  took  the  opportunity 
to  criticize  constructively  the  factors  when  given  the  opportunity  to  do  so 
in  item  3.  Some  of  the  responses  obtained  from  those  who  were  critical 
of  the  factor  detail  were;  too  complex,  overly  detailed,  armament  prepara- 
tion is  not  distinguishable  from  other  factors,  inadequate  factors,  and  fac- 
tor E (communications)  should  be  separated  into  friendly  and  enemy  com- 
munications. 

Items  4,  5,  and  6 involved  rating  the  factors  along  scales  which 
ranged  from  zero  to  100,  on  naturalness,  understandability,  and  com- 
prehensiveness. The  resultant  means  and  standard  deviations  arc  pre- 
sented by  experimental  group  and  characteristic  in  Table  4-18. 
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Table  4-18 

Mean  (M)  and  Standard  Deviation  (S.  D.  ) of  Ratings  of  Factors 
on  Naturalness.  (Indorstandabilitv,  and  Comprehensiveness 


Experiment  1 Experiment  2 


Characteristic 

iVI 

S.  D. 

M 

, S.  D. 

Naturalness 

59.  8 

18.  3 

42.  2 

20.  1 

Understandability 

66,  2 

19.  3 

50.  9 

23.  2 

Comprehensiveness 

66.  2 

19.  3 

51.2 

27.  5 

The  respective  frequency  distributions  of  responses  are  presented 
in  Tables  4-19,  4-20,  and  4-21. 


Frequency  Distribution 

Table  4-19 

of  the  Ratines 

of  Naturalness  of  Factors 

0-  20 

Experiment  1 

Experiment  2 
3 ■ 

21-  40 

8 

15 

41-  60 

5 

6 

61-  80 

14 

5 

81-100 

1 

Table  4-20 

Frequency  Distribution  of  the  Ratings  of  Understandabilitv  of  Factors 

0-  20 

Experiment  1 

Experiment  2 
2 

21-  40 

4 

10 

41-  60 

4 

8 

61-  80 

17 

8 

80-100 

3 

1 
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. table  4-21 


Frequeticy  i) 


G--  20 
2l--  40 
41 --  60  * 
61^  80 
81^100 
No  Response 


Lstrihution  of  the  Ratinp;s  oF  Comprehensiveness  of  Factors 
Experiment  1 Experiment  2 


Generally,  the  understandability  and  comprehensiveness  of  the  fac- 
tors were  rated  higher  than  the  naturalness.  This  finding  is  difficult  to 
6Pmprehend  because  the  factors  are  derived  from  a multidimensional  scal- 
ing analysis  of  the  message  perceptions  of  intelligence  officers.  Neverthe- 
less, on  the  average,  the  naturalness  was  rated  at  about  the  midpoint  of 
the  scale.  For  the  experiment  1 analysts,  the  understandability  and  com- 
prehensiveness v.'cre  rated  well  above  the  scales'  midpoints  on  the  average, 
While  for  the  experiment  2 analysts,  these  two  characteristics  were  rated 
at  about  the  midpoint.  All  three  Table  4-10.  4-2  0,  and  4-21  frequency  dis- 
tributions indicate  the  ratings  of  the  experiment  1 analysts  to  be  skewed 
towards  the  upper  end.  No  experiment  1 analy.sls  found  the  factors  to  be 
very  unnatural  or  very  difficult  to  understand.  The  skewness  is  not  evi- 
dent in  the  corresponding  data  for  the  experiment  2 analysts.  Again,  the 
Ihereased  difficulty  evidently  experienced  by  the  experiment  2 analysts 
during  their  inquiry  task  seems  to  have  generalized  and  affected  their  rat- 
ings of  the  factors. 


About  one-third  of  the  sample  of  each  experiment  responded  to 
Item  7,  which  asked  for  suggestions  relative  to  other  factors  that  could 
be  added  to  the  tax  nioiny  and  for  suggestions  relative  to  factors  which 
might  be  deleted  witliout  hindering  the  completeness  of  the  factor  set.  The 
Sb'g'geslious  for  additional  factors  included; 


• allied  forces  information  from  front  line  troops 

• terrain  considerations 

• weather  reports 

• special  weapons 

• air  activity 

• nuclear  activity 

• enemy  intelligence  collection 
W personalities 

• 'W-e-apon  systems 


99 


■-mi 


-rrrr 


'i'Wo'  of  the  facfors  wg  re  identif  ied  as  candidates'  for  deletion.  One 
analyst  thought  that  the  Plans  and  Expectations  factor  might  be  eliminated 
on  the  groimd.s  that  it  is  very  close  in  meaning  to  the  class  called  Prepara- 
frohs  and  Readiness.  The  other  factors  nominated  for  deletion  by  each  of 
fwo  subjects  were  Armament  Preparation  and  Type  of  Activity.  No  rea- 
sons were  given  for  these  suggested  deletions. 


Classes 

A parailel  set  of  questions  was  included  relative  to  the  taxonomic 
classes.  An  exeeedinglv  large  proportion  of  the  analysts  of  experiment  1 
regarded  the  classes  as  either  having  the  proper  amount  of  detail  or  as  be- 
ing overly  detailed  (85  per  cent).  Three-fourths  of  the  analysts  in  experi- 
ment 2 were  of  the  same  opinion.  It  appears  that  for  the  vast  majority  of 
the  subjects,  the  classes,  as  they  stand,  incorporate  adequate  detail. 


ifi  § fbiibwup  question  (item  9),  one  analyst  indicated  that  classes 
S3  and  24,  liigh  Reliability  Source  and  l.OW  Reliability  Source,  were  diffi- 
§Ult  to  under-stand.  Two  analysts  suggested  a consolidation  of  classes  13 
(Preparations  and  Readiness)  and  17  (Competence  of  Personnel  and  Equip- 
ffieht),  which  appeared  to  them  to  have  overlapping  meaning.  The  final 
constructive  criticism  was  related  to  class  11,  Actual  and  Threatened  Con- 
tact, The  analyst  indicated  that  additional  examples  could  help  to  explain 
the  class  more  elearly.  He  gave  the  following  examples:  fire  fighting, 
sighting,  hearing,  and  indirect  fire.  On  balance,  it  appears  that  minor 
Class  revisions  along  the  lines  suggested  above  would  make  them  more  ac- 
ceptable to  the  military  personnel  involved  in  the  present  sample. 

The  naturalness,  unacrslandability,  and  comprehensiveness  of  the 
classes  were  evaluated  using  the  same  type  of  magnitude  estimation  scales 
as  for  the  factors.  The  resultant  means  and  standard  deviations  are  pre- 
sehted  in  Table  4^22.  The  respective  frequency  distributions  of  responses 
She  firesented  in  Tables  4-23,  4-24,  and  4-25. 


TOO 


Table  4-22 


I 

1 

Mean  (M)  and  Standard  Deviation  (S.  D.  ) of  Hatings  of  Classes  | 


On  Naturalness, 

I'nderstandabili 

tv.  and 

Comprehe 

nsivenes.‘ 

> 

Experiment  1 

Experiment  2 

Characteristic 

M 

S.  D. 

~n 

S.  D. 

Naturalness 

64.  4 

17.4 

46. 

8 

22.  2 

Dnderstandability 

66.  6 

17.  1 

51. 

0 

24.  9 

Comprehensiveness 

64.  5 

16.  5 

56. 

1 

27.  3 

Table  4 

-23 

• 

i Frequency  Distribution  of  Ratings  of  Naturalness  of  Classes 

h Experiment  1 Experiment  2 


0*  20 

- 

3 

21-  40 

5 

12 

41-  60 

10 

6 

61-  80 

11 

7 

■ 81-100 

2 

1 

•No  Response 

1 

2 

Table  4-24 

Prequenev  Distribution 

of  Ratines,  of  Understandabilitv 

of  Classes 

Experiment  1 

Experimen 

0-  20 

• 

3 

21-  40 

4 

9 

41-  60 

8 

8 

61-  80 

12 

7 

81-100 

4 

2 

No  Response 

1 

2 

A 
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Table  4-25 


Frequency  Distribution  of  Hatin.es  of  Comprehensiveness  of  Classes 


Experiment  1 Experiment  2 

0-20. 

21-  40  3 

41-  60  10 

61-  80  12 

81-100  2 
No  Response  2 


Generally,  the  three  characteristics  considered  were  rated  moderately 
above  the  midpoint  of  the  scale  by  the  experiment  1 analysts  and  at  about  the 
midpoint  by  the  experiment  2 analysts.  The  respective  frequency  distiibutions 
reflect  the  same  trend.  No  experiment  1 analyst  rated  the  classes  as  very 
unnatural,  very  difficult  to  understand,  or  fully  uncomprehensive.  Yet, 
three  or  four  of  the  experiment  2 analysts  considered  the  classes  to  possess 
these  extreme,  negative  characteristics.  About  half  of  the  analysts  of  both 
groups  rated  the  classes  above  60  on  all  three  characteristics.  All  of  this 
seems  to  suggest  that  the  taxonomic  classes  possess  a moderate  degree  of 
acceptability  but  that  further  refinement  would  possess  value  from  the  point 
of  view  of  fuller  reputability.  Yet,  there  is  nothing  in  the  current  data  which 
suggests  that  the  class  axis  of  the  developed  taxonomy  has  not  at  least  partial- 
ly achieved  its  objectives. 


4' 

4 

7 

9 

4 
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Evaluation  of  the  Svstem 

Items  1 and  14  of  the  opinion  questionnaire  were  open  ended  in  nature 
and  asked  about  the  easiest  and  most  difficult  aspects  of  the  system  and  for 
general  comments.  There  were  mixed  opinions  relative  to  the  easiest  and 
most  difficult  aspects  of  the  system.  While  some  indicated  that  they  found 
it  easy  to  classify  the  messages  into  classes  and  difficult  to  classify  them  in- 
to factors,  about  an  equal  number  found  the  opposite  to  be  the  case.  Very 
few'  analysts  said  that  the  system  was  difficult  to  use.  The  nature  of  the 
difficulties  found  with  the  system,  as  identified  by  the  analysts  m response 
to  item  1,  consisted  of  the  following:  classifying  a message  into  a single  cate- 
gory (factor/ class),  understanding  the  definitions,  learning  a new  vocabulary, 
and  being  able  to  distinguisli  between  factors  and  classes. 


Additional  comments  received  in  response  to  item  14  did  not  indi- 
cate an  unworkable  or  unacceptable  system.  A small  number  of  analy.sts 
indicated  that  they  would  have  liked  more  detailed  definitions.  Some  anal- 
ysts were  of  the  opinion  that  certain  factors  and  classes  were  not  com- 
pletely distinguishable  from  others  and  that  adding  more  detail  to  the  defi- 
nitions might  correct  this  situation.  One  analyst  claimed  that  the  system 
was  too  extensive  for  daily  use  and  would  be  easier  to  use  if  it  was  made 
simpler. 

In  summary,  while  differences  existed  among  the  overall  means 
of  both  groups  on  the  ratings  of  naturalness,  understandability,  and  com- 
prehensiveness of  the  derived  system,  the  ratings  were  generally  above 
or  at  the  midpoint  of  a 100  point  scale.  This  suggests  that  a generally  ac- 
ceptable system  is  on  hand.  Additional  and/or  imprdved  training,  diction- 
ary enhancement,  and  refinement  of  the  taxonomic  system  might  serve  to 
augment  the  acceptability  of  the  system. 


CHAPTER  V 


MODEL  INTEGRATION  AND  FUNCTIONAL  DESCRIPTION  OF 
COMPUTER  IMPLEAIENTATION  OF  THE  BATTLEFIELD 
MESSAGE  CLASSIFICATION  SYSTEIU 


The  prior  chapters  of  the  present  report  described  the  methods 
and  results  of  work  oriented  towards  the  development  of  a taxonomy 
■which  describes  the  battlefield  language  structure.  While  the  derived 
taxonomy  was  suggested  to  be  adequately  general  and  useable  by  battle- 
field analysts  for  message  classification  and  retrieval  purposes,  the 
opinion  questionnaire  (Chapter  IV)  indicated  concern  by  some  analysts  that 
the  model  was  o-verarticulated  in  some  aspects  and  underarticulated  in 
other  aspects.  Accordingly,  a total  model  review  session  was  conducted 
at  the  U.  S.  Army  Intelligence  Center  and  School,  Fort  Huachuca.  The  pur- 
pose of  the  review  was  to  acquire  the  insights  of  e.xperienced  intelligence 
personnel  relative  to  areas  of  needed  taxonomy  extensio.n  and  consolida- 
tion. The  review  was  conducted  informally  and  was  performed  by  14  offi- 
cers (including  one  foreign  military  liaison  officer)  and  senior  enlisted 
personnel.  All  were  currently  active  in  some  phase  of  battlefield  infor- 
mation analysis. 

The  comments  of  these  panel  members  were  .in  general  conform- 
ity with  the  indications  of  the  Chapter  IV  opinion  questionnaire  results.  On 
the  basis  of  these  comments,  the  taxonomy  was  revised  and  set  in  a re- 
vised current  form.  This  revised  taxonomy  is  called  the  "final  model"  in 
subsequent  aspects  of  this  report. 


The  Final  Model 

The  final  model  is  also  conceived  as  a matrix  with  factors  forming 
one  axis  and  classes  forming  the  second  axis. 

The  factors  of  the  final  model  are  the  same  as  in  Table  3-1  of  this 
report.  However,  the  number  of  classes  is  reduced  considerably.  The 
classes  removed  and  the  associated  reasoning  are  presented  in  Table  5-1. 
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Additionally,  four  classes  were  added; 

Terrain 
Weather 
Special  Weapons 
Air  Activity 

Accordingly,  the  classes  in  the  final  model  are  those  presented 
in  Table  5-2. 

Table  5-2 


Classes  in  Final  ATodel 


Composition  of  units 

Markings  and  insigna 

Position  of  forces  and  equipment 

Strength  of  units 

Replacements 

Weapons  and  equipment 

Analysis 

Characterization  of  missions 
Sabotage 

Strategy  and  tactical  doctrines 
•Preparations  and  readiness 


Chronology 

Information  about  officers 

Competence  of  personnel  and  equipmen 

High  reliability  source 

Conditional  phrases 

Unknovm  information 

Terrain 

Weather 

Special  Weapons 
Air  Activity 
Other 
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Functional  Description  of  Computer  Implementation 
of  the  Battlefield  Message  Classification  Syst  e m 


As  pointed  out  by  Finley,  Muckier,  Gainer,  and  Obermayer  (1975), 
the  development  of  a taxonomy  which  describes  the  natural  order  of  things 
is  not  a sufficient  end,  in  itself,  to  the  practitioner.  In  the  present  case, 
the  taxonomy  was  developed  in  order  to  make  evident  those  battlefield  lang- 
uage structure  properties  which  chould  be  employed  as  a basis  for  increas- 
ing battlefield  information  flow  and  intelligence  emphasis.  To  this  end,  a 
functional  design  was  derived  of  a system  for  processing  battlefield  mes- 
sages using  a general  purpose  digital  system  and  the  derived  taxonomy. 

We  conceive  of  a message  processing  system  composed  of  a network 
of  typewriter/ display  terminals,  manned  by  combat  (fnilitary)  personnel  who 
represent  both  the  sources  of  input  messages  for  storage  as  well  as  by  in- 
telligence analysts  who  refer  to  and  extract  correlations  from  such  mes- 
sages. The  network  of  terminals  is  deployed  in  a battlefield  (or  test)  situ- 
ation. The  computer  may  be  interfaced  to  the  network  through  a variety  of 
means.  These  are  not  of  particular  im.portance  here.  Figure  5-1  presents 
such  a potential  system  for  illustrative  purposes. 

The  functional  capabilities  of  the  system  given  in  the  next  section 
include  storage  of  messages  and  message  headers  by  class(es)  and  factor(s), 
a flexible  user  syntax,  including  inquiries,  security  features,  automatic 
.purging  for  archival  storage,  and  user  displays  for  teaching  the  operation- 
al procedures. 

A block  diagram  of  the  program  is  presented  in  Figure  5-2,  which 
shows  the  overall  logic  flow  between  the  program  modules.  The  composi- 
tion of  files  to  be  stored  is  presented  in  Table  5-3, 

Of  particular  importance  in  defining  such  a terminal  oriented  sys- 
tem is  specification  of  the  types  of  input  messages  'which  a user  will  have 
at  his  disposal.  A brief  list  of  the  types  of  transactions  considered  is  given 
in  Table  5-4.  These  capabilities  are  further  detailed  in  Appendix  D in  the 
form  of  user  syntax  diagrams  for  each  transaction  type  (input,  maintenance 
of  the  store  of  messages  and  the  networks,  inquiries,  status  display  re- 
quests, and  user  orientation  (teaching)  capabilities. 


ONE  OR  MORE 

KEYBOARD 

TERMINALS 

WITH  SMALL 

DISPLAYS  FOR 

FORWARD 

OBSERVERS 


INPUT  AND 
TEACH 

TRANSACTIONS 

ON^Y 


ONE  OR  MORE 
CRT 

TERMINALS 

FOR 

INTELLIGENCE 

ANALYSTS 


SECURE  COMPUTER  CENTER 

(VAN  MOUNTED  FOR  BATTLEFIELD  DEPLOYMENT  OR 
COMMERCIAL  EQUIPMENT  FOR  REAR  AREA  DEPLOYMENT) 


''  FRONT  END 

CENTRAL 

MAIN 

't  PROCESSOR(S) 
/ 

■ PROCESSOR(S) 

MEMORY 

CONTROLLERS  FOR  PERIPHERAL  DEVICES 


LINE 

PRINTER 


MAGNETIC 
TAPE  FOR 
MESSAGE 
ARCHIVES 


ALL 

TRANSACTIONS 

EXCEPT 

MAINTENANCE 


LOCAL  CRT 

TERMINALS  FOR  SYSTEM 
TEST  AND  INPUT  OF 
MESSAGES 

(ALL  TRANSACTIONS) 


REMOTE  VIA  TELECOMMUNICATIONS 
LOCAL  VIA  DIRECT  LINES 


CRTS  FOR 
SECURITY 
ADMINISTRATOR 
AND  SYSTEM 
CONTROL 


FIGURE  5-1.  FUNCTIONAL  EQUIPMENT/NETWORK  DIAGRAM. 
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START 


transaction 

TYPE  / 


<NPUT 

INQUIRY 

MAI  NT 

1 

DISPLAY 

1 

1 

TEACH 

— - « 

PROCESS  INQUIRY 
TRANSACTION 

INQUIRY 


PROCESS  DISPLAY 
TRANSACTION 


DISPLAY 


PROCESS  TEACH 
TRANSACTION 


TEACH 


FIGURE,  ‘r- 2-.  GLOBAL  FLOW  CUAF-IT 
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Table  5-3 


THE 

NAME 

Termi na I 


Message 


Log 


CLASSES 

FACTORS 

TEACH 


File  Composition 


ONE  RFCORP 

CONTENTS  FOR  EACH 

terminal  code  terminal 

on/or f status 

last  time/datc  used 

highest  authorized  security  level' 

location  code 

list  of  authorized  user  codes  by  transaction 
type 

senders  time/date 
senders  ID  ■ 
receivers  ID 

encoding  indicator  header 

message  type 

message  pri ori ty 

message  reliability 

message  security  level 

message 

A.  Traffic  Data  Day 

no.  of  messages  by  terminal  by 
transaction  type  during  current 
(non  archival)  period 

B.  Security  Violation  Data 

no.  of  violations  by  transaction 
type  and  by  terminal 

C.  Statistical  Data 

no.  of  message  by  priority, 
security  level 

no.  of  messages  by  transaction  type 
by  terminal  by  N 

no.  of  frequency  distribution  of  me 
messages 

Names  & Descriptions 
Names  & Descriptions 


C lass 
Factor 

Transaction  Type 


Tabic  5-4 


fvn 

(MPUT 

MAINTENANCE 

INQUIRY 

DISPLAY 

TEACH 


Transaction  Typos 
PURPOSE 

Submit  of  messages  to  the  system 

Maintain  tne  network  includes:  adding,  deleting,  changing 
security  codes  of  terminals,  displays  of  security 
violations,  transfering  old  messages  to  archives 

Find,  count,  display  the  number  or  content  of  messages 
meeting  specified  search  conditions 

Display  classes,  factors,  message  statistics,  location, 
security,  or  traffic  for  terminals  on  request 

Ihform  user  on  procedures  to  be  used  in  performing  the 
types  of  system  transactions 
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Functional  Charac  t e r i s t i c s 


The  description  of  the  functional  characteristics  is  divided  into 
two  parts.  The  first  part  describes  initial  capabilities.  The  second 
part  describes  enhanccment.s  that  can  be  added  to  the  system  as  desired. 


Initial  Functional  Canat)ilities 

The  system  possesses  three  major  functions: 

1.  message  input 

2.  message  storage 

3.  message  and  statistics  retrieval 

The  additional  requirements,  SECURITY  and  RELIABILITY/ RECOVER - 
ABILITY,  are  embedded  in  these  major  functions.  A separate,  but  im- 
portant, additional  function  is  MAIYTENAXCE  of  the  terminal  network 
and  system.  Consider  first  the  individual  requirements  of  message  in- 
put, storage,  retrieval,  system  maintenance,  and  reliability  separately. 
Security  is  discussed  within  each  discussion. 


Message  Input 

The  system  must  allow  many  users  to  enter  simultaneously  com- 
plete messages  including  message  te.'ct,  headers,  and  the  category(ies)  in 
Which  each  message  is  to  be  placed  (this  categorization  could  conceivably 
be  performed  automatically).  While  the  average  message  is  short  (would 
fit  on  one  CRT  terminal  screen),  the  system  must  be  able  to  handle  much 
longer  messages.  The  system  must  be  flexible  and  forgiving  of  user  er- 
rors. It  must  be  able  to  accept  messages  from  all  levels  of  experience. 

It  must  prom[)t  the  novice  user  for  each  piece  of  information  required  but 
allow  the  experienced  user  to  enter  all  data  without  time  wasting,  extrane- 
ous communication.  It  must  accept  the  elements  of  the  header  in  any  or- 
der. No  message  should  be  accepted  by  the  system  until  the  security  of 
the  User  and  the  security  of  tlie  terminal  have  been  verified.  Any  security 
Violations  should  be  reported  to  the  security  officer  immediately  and  logged. 


Message  Storage 

The  .system  must  be  able  to  store  messages  at  two  levels:  immedi- 
ate access  and  archival.  Additionally,  the  system  should  allow  groups  of 
ittessage.s  to  lie  moved  from  one  level  to  anotlier,  as  described,  without 
affecting  the  alulily  to  store  incoming  messages. 
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Retrieval 


The  system  must  allow  a small  number  of  retrieval  stations  to  ac- 
cess simultaneously  stored  messages  in  user-oriented  conversational  lan- 
guage. There  are  two  levels  of  inquiries:  those  which  ask  for  actual  mes- 
sage text  and  those  which  ask  for  statistics  about  groups  of  messages. 

Before  allowing  any  retrieval  transactions,  the  system  must  veri- 
fy that  both  the  station  (terminal)  and  user  are  cleared  for  inquiry  trans- 
actions for  text  or  statistics  as  appropriate.  In  addition,  before  any  mes- 
sage text  can  be  transmitted  to  a station,  the  clearance  of  the  station  and 
user  must  be  as  high  as  the  security  level  of  the  message. 

The  user  must  be  able  to  form  a set  of  messa'ges  selected  by  one 
or  more: 

a.  category 

b.  class 

c.  factor 

d.  date 

e.  security 

f.  reliability 

g.  sender 

h.  addressee 

i.  key  word  or  phrase 

The  user  should  be  able  to  combine  selected  sets  of  messages  and  be  able 
to  form  subsets  of  selected  messages  by  further  refining  the  selection  cri- 
teria. 


For  any  selected  set  of  messages,  the  system  should  be  able  to  dis- 
play the  message  text  or  display  statistics  on  the  messages  in  the  set.  The 
available  statistics  niust  include  simple  message  counts  and  tabulations  or 
plots  or  other  visual  display  of  the  relationship  between  pairs  of  selection 

variables. 

Messages  and  statistics  should  be  displayed  on  the  user's  terminal 
(CRT)  and/or  printed  by  a high  speed  printer. 

A complete  set  of  "TKACIl"  functions  must  be  provided  as  an  aid  to 
the  user. 
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Maintenance 


Since  the  maintenance  functions  directly  affect  the  security  and  reli- 
ability of  the  system,  these  functions  must  be  performed  only  froin  within 
the  physically  secure  confines  of  the  main  computer  facility. 

The  maintenance  functions  should  include  at  least: 

a.  add  or  delete  a terminal  from  the  network 

b.  modify  the  security  level,  functional  cap- 
abilities, and  valid  users  of  a terminal 

c.  modify  the  user's  security  or  capabilities 

d.  monitor  the  network,  both  in  real  time  and  by 
placing  all  traffic  to  and  from  a set  of  termi- 
nals into  a log 

e.  force  archival  storage  or  recall  of  any  selected 
set  of  messages 

Reliability 

Since  the  proposed  system  is  not  part  of  a tactical  message  trans- 
mission system,  it  does  not  need  to  be  as  reliable  as  a store-and-forward 
message  processing  system.  The  reliability  levels  of  commercially  avail- 
able computer  systems  should  be  acceptable  for  present  needs.  However, 
the  system  must  have  recoverability  features  to  minimize  operator  and  user 
confusion  after  a system  failure. 

The  functions  that  need  to  be  performed  after  a system  failure  are: 

a.  force  each  user  to  log  on  before  receiving  or 
inserting  any  further  data 

b.  if  an  entire  transaction  was  received  from  a 
user  (but  was  not  completely  processed),  re- 
start the  processing  for  that  transaction 

c.  if  only  part  of  a transaction  was  received  from 
a user,  show  the  user  what  was  received  and 
ask  for  the  remainder 

d.  report  any  unrecoverable  situation  to  operations 
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Enhancements 


Two  primary  areas  exist  for  additional  enhancement  and  sophisti- 
cation: the  input  phase  and  the  inquiry  phase. 

In  the  area  of  message  input,  the  system  could  recognize  actual 
tactical  message  formats.  This  would  allow  message  input  to  be  essenti- 
ally a clerical  task  and  to  be  separated  from  the  task  of  category  assign- 
ment. Messages  could  then  be  input  by  clerk  typists,  optical  scanners, 
paper  tape,  or  a direct  connection  with  a tactical  automatic  message  proc- 
essing system.  Once  a message  has  been  received  by  the  system,  it  could 
be  displayed  on  a CRT  terminal  for  actual  analysis  and  category  assign- 
^ ment.  These  enhancements  are  easily  within  current  technology.  The  next 

enhancement  could  be  to  have  the  system  itself  read  and  analyze  each  mes- 
sage as  received,  and  assign  message  categories.  This  enhancement  pos- 
l sesses  some  technical  risk  for  arbitrary  messages. 

m. 

The  second  area  of  possible  enhancement  is  related  to  message  in- 
quiry. The  class  of  possible  questions  could  be  made  more  general.  The 
possibility  exists  that  the  system  can  be  enhanced  so  that  it  will  suggest  an 
"answer"  or  suggested  course  of  action  relative  to  a query.  Such  a "deci- 
sion recommending"  feature  would  require  research  into  the  "decision  pol- 
icy" of  current  analysts.  However,  such  "policy  capturing"  has  been  demon- 
strated to  be  entirely  feasible  (Slovic  & Lichtenstein  1970;  Dawes,  1970; 
Christal,  1963;  Stephanson  & Ward,  1971). 

Similarly,  the  system  could  be  enhanced  to  perform  predictions  and 
trend  analysis.  Predictions  could  be  made  relative  to  presumed  enemy  goals 
and/or  tentative  friendly  actions.  Answers  could  be  based  on  a "reliability 
factor"  to  insure  that  only  "best"  answers  are  provided.  In  cases  where 
content  is  contradictory,  multiple  answers  could  be  provided  with  assigned 
probabilities. 

The  modular  design  of  the  proposed  system  permits  the  addition  of 
any  of  these  enhancements  in  an  orderly  fashion  without  affecting  existing 
capabilities.  This  design  also  permits  other  kinds  of  functions  to  be  added. 
Largo  size  color  CRT's  could  be  added  to  show  terrain  and  weather  with 
past,  current,  or  predicted  future  troop  positions. 

These  suggested  enhancements  show  possible  system  expansion. 

Some  of  these  features  can  be  readily  implemented  at  this  time.  Others 
would  require  some  effort  for  full  implementation.  All  of  these  features 
can  be  performed  on  currently  existing  hardware,  and  most  features  have 
been  demonstrated  to  some  extent  within  the  current  software  state  of  the 
art. 
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The  program  wovild  be  implemented  as  a transaction  processor 
system  and  is  divided  into  three  major  components:  (1)  central  transac- 
tion processor,  (2)  modules  that  process  individual  transactions,  and  (3) 
auxiliary  routines. 

The  central  transaction  processor  and  global  system  flow  are  pre- 
sented in  Figure  5-2.  The  central  transaction  processor  consists  of  four 
modules:  INITIAL,  READ,  TERMINAL  STATUS,  and  VALIDATE. 


Module  1:  INITIAL 


The  INITIAL  module  performs  system  initialization  cither  at  system 
Startup  or  after  a system  failure.  It  initializes  all  data  areas  and  checks 
for  existing  copies  of  files.  In  the  event  of  a restart  after  a system  failure, 
INITIAL  calls  for  the  RECOVER  module  (#22).  INITIAL  marks  all  termi- 
nals as  active  or  inactive  and  logged  off.  When  all  initialization  and  recov- 
ery actions  are  complete,  control  is  passed  to  the  READ  module. 

Modules  2,  3,  and  4 form  a closed  central  loop.  Controlled  by  this 
loop,  but  as  separate  tasks,  are  variable  number  of  copies  of  the  processing 
modules.  Each  copy  of  a processing  module  completely  processes  one  re- 
quest at  a time.  When  completed,  it  checks  for  another  request  in  its 
queue.  If  present,  it  processes  that  request.  If  its  queue  is  empty,  the 
module  will  wait  for  a short  period  and,  if  its  queue  remains  empty,  it  will 
be  discontinued.  This  feature  frees  system  resources  and  helps  to  ensure 
that  the  system  is  always  balanced  for  the  current  mix  of  requests. 


Module  2;  READ 

The  READ  module  accepts  a transmission  from  the  data  communi- 
cations queue  for  any  station,  writes  the  transmission  to  the  recovery  tape 
With  date,  time,  and  sender  information,  and  then  checks  the  transmission' s 
content.  If  the  transmission  is  a log-on  or  log-off  request,  the  transmission 
Is  passed  to  the  TERMINAI.  STATUS  module  (#3).  If  the  transmission  is  not 
a log-on  or  log-off  request,  the  terminal  file  is  checked  to  ensure  that  the 
Sending  terminal  is  logged  fm.  If  not,  a security  error  is  reported  and  the 
message  droppt'd.  If  tlu'  transmission  is  the  first  (lart  of  a new  transaction, 
it  is  passed  to  the  VAI.lDATi:  module  (#4).  If  the  transmission  is  a continu- 
ation of  a currimt  in  process  transaction,  the  transmission  is  queued  to  the 
particular  copy  of  a processing  module  which  has  started  processing  that 
transaction. 


Module  3:  TERMINAL  STATES 


The  TERMINAL  STATES  module  handles  all  log-on  and  log-off  activ- 
ity. If  a log-on  request,  the  module  verifies  that  the  user  is  a valid  user 
of  that  terminal.  If  not,  a security  error  is  reported,  and  all  further  activ- 
ity from  that  terminal  is  ignored  until  the  security  officer  takes  appropriate 
action.  In  any  event,  the  terminal  file  is  updated  to  show  the  latest  status 
of  the  terminal. 

Module  4:  VALIDATE 


The  VALIDATE  module  performs  initial  validation  on  transactions. 
It  verifies  that  the  specified  transaction  is  valid  for  the  specific  user.  Any 
errors  are  reported  to  the  user  and  are  logged.  If  the  transaction  is  vmlid, 
it  is  queued  for  one  of  the  processing  modules  on  the  basis  of  its  type.  If 
an  idle  copy  of  a program  module  of  the  required  type  is  available,  trans- 
action is  given  to  that  copy.  If  no  idle  module  copy  is  available,  the  trans- 
action may  be  queued  to  a busy  module  to  be  processed  when  that  module 
completes  its  current  task,  or  a new  copy  of  a module  may  be  initiated. 
This  decision  is  based  on  number  of  queued  transactions,  number  of  active 
processing  modules,  message  priority,  and  expected  and  desired  response 
times.  Once  the  transaction  has  been  queued  to  a processor  module,  con- 
trol returns  to  the  READ  module  (#2). 


Module  5:  INPUT 

The  INPE'T  module  processes  all  message  input.  It  verifies 
header,  text,  and  category  information.  If  no  errors  are  detected,  the 
message  is  stored  in  immediate  message  storage.  Keys  are  built  for 
specified  categories  and  header  information  for  quick  access.  If  an  er- 
ror is  detected  on  input,  the  user  is  lead  to  the  correct  response  by  a 
series  of  prompts.  Successful  storage  of  the  message  is  reported  to  the 
user  and  logged  (P'igurc  5-3). 


Module  6:  MAINTENANCE 

The  MAINTENANCE  module  validates  maintenance  requests  and 
reports  errors  to  the  user  by  means  of  prompts  (Figures  5-4).  The  vali- 
dated request  is  processed,  given  to  the  user,  and  logged. 
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SET  UP  DEFAULTS 


store  header  and 

MESSAGE  IN  MESSAGE  FILE 
BY  CATEGORY(IES)  SPECIFIED 
AS  CODED 


CONFIRM 
COMPLETION  OF 
TRANSACTION  TO 
USER 


VALIDATE 


STORE 


CONFIRM 


ERROR 


GENERATE 
APPROPRIATE  ERROR 
MESSAGES  AND 
PROMPTS 


PROMPT 


FIGURE  5--3.  INPUT  MODULE  FLOW 
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FIGURE  5-4.  MAINTENARCF  MODULE  FLOW 
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Module  7:  INQUIRY 


The  INQUIRY  request  module  validates  any  errors  corrected  by 
means  of  prompts  (Figure  5-5).  The  request  is  processed  (this  may  in- 
volve sequential  searching  of  a large  number  of  messages),  and  the  final 
formatted  results  are  sent  to  the  requester.  If  the  request  is  expected  to 
take  a time  which  exceeds  a predetermined  threshold,  the  sender  is  noti- 
fied that  his  request  is  in  process.  Before  actual  message  text  can  be 
sent  to  the  user,  the  security  of  the  user  and  terminal  are  checked.  The 
request  is  logged  as  completed  when  the  user  has  received  his  requested 
information. 


Module  8:  DISPLAY 

The  DISPLAY  module  validates  a request  and  any  errors  are  cor- 
rected by  means  of  prompts  (Figure  5-6).  The  requested  information  is 
returned  to  the  user,  and  the  completion  of  the  request  is  logged. 


Module  9:  TEACH 


The  TEACH  module  validates  a request  and  the  appropriate  informa- 
tion is  displayed  to  the  user  from  the  teach  file  (Figure  5-7).  The  request 
is  logged  when  complete. 

There  are  three  auxiliary  modules;  ERROR,  LOG,  and  RECOVERY. 
These  are  used  for  special  functions  by  other  modules. 


ERROR 

The  ERROR  module  is  used  to  report  syntax  errors  to  the  user  and 
security  errors  to  the  security  officer.  All  errors  are  logged  for  later 
performance  evaluation.  (This  module  is  not  shown  in  the  flow  logic  draw- 
ing since  it  may  be  called  from  any  PROAIPT  or  VALIDATE  module^  ) 


LOG 


The  LOG  module  is  responsible  for  adding  information  to  the  log 
file.  A history  of  all  action  taken  by  the  system  is  contained  in  Hie  log. 
The  receipt  of  each  request  and  its  completion  is  logged  along  with  date, 
time,  and  user  identification.  All  errors  and  maintenance  actions  are 
also  logged,  as  are  all  recovery  actions.  By  appropriate  analysis  of  the 
log,  questions  about  system  or  user  performance  can  be  answered. 
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FIGURE  5-5.  MESSAGE  CACHE  INQUIRY  FLOW 
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FIGURL  5-d.  display  MODULE  FLOW 
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FIGUF^f.  b-'l . TEACH  MODULE  FLOW 


RECOVERY 


The  RECOVERY  module  is  called  only  after  a system  failure.  By 
reading  the  recovery  tape  and  matching  each  transaction  on  the  tape  with 
the  log,  the  status  of  each  transaction  at  the  time  of  system  failure  can  be 
determined  and  appropriate  action  taken. 


CHAPTER  VI 


' DISCUSSION  AND  CONCLUSIONS 


The  present  study  was  instituted  to  determine  whether  or  not  a nat- 
ural order  could  be  derived  for  battlefield  language.  Such  a natural  order 
was,  in  fact,  shown  to  be  possible.  It  was  possible  to  derive  a model 
which  is  based  on  two  complementary  logics. 

The  first  logic  (multidimensional  scaling)  was  based  on  derivation  of 
the  factorial  structure  of  the  perceptions  of  battlefield  analysts  of  battlefield 
I language.  The  isolated  factors  form  one  axis  of  the  final  model.  This  axis, 

which  might  be  called  the  perceptual  axis,  is  rooted  in  the  language  percep- 
tions of  battlefield  analysts. 

Ip.  The  second  logic  was  based  on  aoplication  of  concepts  of  transforma- 

. tional  grammar.  The  results  of  this  analysis  yielded  the  second  axis 

(classes)  of  the  final  model.  This  second  axis  might  be  called  a linguistic 
axis.  The  linguistic  axis  is  based  on  the  structural  aspects  of  battlefield 
language. 


i 

i 

t 

i 

fr 

!» 

I 


Accordingly,  it  can  be  argued  that  the  final  model  is  compatible 
with  both  the  actual  battlefield  language  structure  and  the  perceptions  of 
battlefield  language  of  intelligence  analysis.  As  suggested  by  Halpin 
(1975),  use  of  such  a model  in  a computer  oriented  message  management 
system  would  make  the  equipment  system  compatible  with  the  language 
and  the  perceptions  of  the  user.  Such  an  approach  seems  entirely  ra- 
tional from  the  integrated  man-equipment  system  point  of  view.  W ithin 
such  a design  construct,  the  information  management  system  is  based  on 
the  language  and  perceptions  of  the  user  rather  than  the  reverse  situation 
in  which  the  user  must  adapt  to  machine  requirements.  Unfortunately, 
this  latter  design  approach  is  the  one  found  in  most  information  manage- 
ment systems. 

It  is  possible  also  that  the  high  obtained  useability  (reliability)  of 
the  derived  taxonomy,  as  evidenced  by  the  two  verificational  experiments 
is  a result  of  the  nalurality  luiilt  into  the  taxonomy.  In  view  of  the  only 
limited  training  giv'en  to  the  participants  in  tiie  two  verificational  experi- 
ments, tlic  useability  indications  were  iiighcr  tlian  might  have  been  an- 
ticipated. While  other  explanations  of  this  finding  are  possible,  the  nat- 
urality  concept  represents  an  attractive  explanatory  construct  and  one 
which  supports  the  utility  of  the  final  model. 
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The  high  obtained  degree  of  congruency  between  the  factorial  struc- 
ture of  the  four  military  intelligence  analysts  whose  perceptions  formed 
the  basis  for  the  four  separate  multidimensional  scaling  analyses  might 
also  have  been  anticipated.  The  meaning  and  interpretation  of  words, 
phrases,  and  sentences  is  learned.  This  holds  whether  a total  language 
or  a sublanguage  is  involved.  The  military/battlefield  subculture  may 
serve  to  crystalize  the  battlefield  language  perceptions  just  as  any  other 
subculture  fosters  common  language  percepts  within  the  subculture's 
members.  For  example,  the  sentence  "He  is  spaced  out"  would  prob- 
ably be  commonly  perceived  by  an  American  teenage  population  but  may 
be  perceived  differently  by  Russian  teenagers.  Unfortunately,  the  pres- 
ent study  did  not  include  a nonmilitary  control  group.  Factorization  of 
the  percepts  of  control  analysts  of  the  same  stimulus  message  set  would 
be  of  interest  relative  to  the  homogenization  conjecture. 

Certainly  other  tests  of  the  utility  and  veridicality  of  the  taxonomy 
are  indicated.  The  present  tests,  as  reported  in  Chapter  IV,  were  based 
on  abbreviations  of  the  total  taxonomy  in  its  form  at  that  time.  The  ques- 
tion of  whether  or  not  the  results  would  still  obtain  when  the  total  taxonomy 
is  employed  remains  open.  However,  we  note  that  the  final  taxonomy  rep- 
resents considerable  consolidation  to  reduce  ambiguity.  Such  ambiguity 
reduction  would  be  anticipated  to  increase  utility.  Accordingly,  there  are 
grounds  for  believing  that  experimental  verification  of  the  final  model 
would  yield  results  wiiich  are  largely  in  conformity  with  the. two  verifi- 
cational  studies  which  were  completed. 

Finally,  it  seems  that  the  battlefield  information  processing  and 
intelligence  derivation  technology  will  eventually  move  towards  a com- 
puterized approach  such  as  that  described  in  Chapter  V.  Such  a system 
would  lead  towards  the  ability  to  perform  online  historical  analysis,  deri- 
vation of  most  probable  enemy"  intent,  trend  analysis  and  prediction,  and 
computer  suggested  course  of  action. 
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Conclusions 


In  view  of  the  data  presented  in  earlier  chapters  of  the  present 
report,  the  following  conclusions  seem  warranted: 

1.  For  the  intelligence  analyst  sample  (and  battlefield 
messages  involved),  the  perceptual  structure  may  be 
described  by  15  orthogonal  factors.  These  factors 
seemed  common  across  the  four  sample  members 
for  whom  separate  multidimensional  scaling  analy- 
ses were  performed. 

2.  From  the  linguistic  point  of  view,  the  structure  may 
be  described  by  a limited  set  of  classes. 

3.  The  combination  of  perceptual  factors  and  linguistic 
classes  into  a taxonomy  yielded  a product  which  ap- 
pears to  be  useful  from  both  the  information  categori- 
zation and  the  information  retrieval  points  of  view. 
Analysts  were  able  to  classify/ retrieve  messages 
relative  to  the  taxonomy,  and  analysts  were  sensi- 
tive to  and  confident  in  the  quality  of  their  perform- 
ance when  employing  the  taxonomy. 

4.  It  appears  as  if  a reasonable  start  has  been  made 
toward  establishing  how  battlefield  messages  have 
attributes  in  common  and  attributes  which  are  dif- 
ferent. 
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INTRODUCTION 


I . 


This  dictionary  describes  a preliminary  system  for  classifying 
battelfield  language.  Basically,  the  system  involves  technique  for 
classifying  battlefield  messages  into  categories  according  to  two 
classification  schemes.  These  two  schemes  each  partition  the 
whole  range  of  battlefield  language.  One  scheme  consists  of  15 
divisions;  each  of  these  is  called  a "factor."  The  second  scheme 
consists  of  27  divisions;  each  of  these  is  called  a "class." 

Within  the  system,  a battlefield  message  may  be  classified 
using  both  classes  and  factors  in  combination  (called  categories). 
Retrieval  of  information  from  this  system  is  accomplished  by 
selecting  for  review  all  information  in  one  or  several  categories. 

This  Dictionary  provides  all  the  basic  information  required 
to  use  this  system.  Sections  II  and  III  list  and  define  the 
factors  and  classes.  Section  IV  consists  of  a model  implementa- 
tion of  the  system.  This  model  serves  as  a reference  which  may 
be  consulted  if  the  use  of  any  category  is  unclear.  For  the  purposes 
of  showing  further  use  of  the  system,  two  analyses  of  the  phrases 
shown  in  the  model  are  presented  in  sections  V and  VI.  Section  V 
alerts  the  user  to  synonyms  in  the  battlefield  language.  Section 
VI  contains  an  alphabetical  list  of  all  the  words  used  in  the 
model  and  defines  these  words  in  terms  of  categories  in  which  they 
are  found. 
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LIST  OF  FACTOR  AND  CLASS  NAMES 


Factor  Names 

A.  Movement  and  Capability  for  Movement 

B.  Numbers,  Trends,  and/or  Source 

C.  Type  of  Activity 

D.  Training,  Morale,  and  Attitudes 

E.  Communications 

F.  Armament  Preparation 

G.  Small  Arms  Capability 

H.  Strength  and/or  Location  of  Units 

I.  Plans  and  Expectations 

J.  Extra  Military  Information  Sources 

K.  Deserter  Information 

L.  Logistics 

M.  Unique  Actions 

N.  Contents  of  Documents 

O.  Civilian-Military  Relationships 
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Class  Names 


1.  Composition  of  Units 

2.  Markings  and  Insignia 

3.  Position  of  Forces  and  Equipment 

4.  Strength  of  Units 

5.  Replacements 

6.  Quantitative  Indications 

7.  Weapons  and  Equipment 

8.  Analysis 

9.  Characterization  of  Missions 

10.  Sabotage 

11.  Actual  and  Threatened  Contact 

12.  Strategy  and  Tactical  Doctrines 

13.  Preparations  and  Readiness 

14.  .Movement 

15.  Chronology 

16.  Information  about  Officers 

17.  Competence  of  Personnel  and  Equipment 

18.  Mental  and  Physical  Condition  of  Troops 

19.  Civilian-Military  Relations 

20.  Sources/Methods  of  Obtaining  Information 

21.  Informer  Source-.Mi  1 itary 

22.  Informer  Source-Non  Military 

23.  High  Reliability  Source 

24.  Low  Reliability  Source 

25.  Conditional  Phrases 

26.  Unknown  Information 

27.  Other 


147 


Ill  . FAPTOR  AX'!)  ri.ASS  1)1:F  I I ’i  1 0\S 
Factor  Dofinilions 


A.  Movomont  and  ratKihilitv  for  Movomont  — relates  to  various 
aspects  ot  movement  oi'  vehicles  :ir,d  men  and  the  capability 
for  such  movement.  Examples  cf  this  type  of  information 
are;  the  lay ing  and  support  capabil.ty  of  pontoon  structures, 
the  actual  movement  of  a troop  unit  or  an  aggressor  vehicle, 
or  the  speed  of  such  movement  and  the  capability  for, 
amphibious  movement. 

B.  Numbers,  Trends,  and  Source--ref ers  to  the  numerical 
aspects  of  units,  supplies,  personnel,  events,  etc.  and 
trends  in  such  aspects  (e.g.,  number,  volume  increases  or 
decreases).  This  factor  also  includes  data  related  to 
the  reported  source  of  information. 

C.  Type  of  Activity — includes  descriptions  of  the  character  of 

various  types  of  operations,  such  as:  reconnaissance, 

patrol,  sabotage,  and  exchange  of  fire. 

D.  Training.  Morale,  and  Attitudes--deals  with  the  training, 
moti vat  ion/morale , and  attitudes  of  the  hostile  forces  and 
with  information  which  bears  on  these.  Examples  are; 

the  disclosure  of  courts  martial,  officer  training  level, 
and  special  training. 

E.  Commun icat ions — includes  information  related  to  field 
communications  and  the  capability  for  such  communication. 

F.  Armament  Preparation — includes  information  concerning 

preparation  of  such  items  as:  fortifications,  heavy  gun 

emplacements,  mine  fields,  and  related  construction. 

G.  Small  Arms  Capability--deals  with  small  caliber  weapons, 
including  their  presence,  strengths,  and  weaknesses. 

H.  Strength  and  Location  of  Units — includes  information  con- 
cerning strength,  striking  power,  and  location  of  units  of 
all  types.  E.xamples  of  these  types  of  information  are: 
the  presence  of  influenza  in  the  aggressor  forces,  arrival 
of  troop  replacements,  type'  and  number  of  support  units, 
and  locational  coordinates. 
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I.  Plans  and  Expectat ions--a  grouping  of  various  types  of 
information  concern  ins  plans,  intent,  and  expectations. 
This  class  includes,  for  example,  commandin';  officer 
statements,  expected  enemy  actions,  and  related  prepara- 
tion . 

J.  Extra  Military  Information  Sources--inc ludes  information 
from  non-military  and  possibly  unreliable  sources  of 
information.  Typifying  this  factor  are  such  sources  as 
civilians  and  civil  broadcasts. 

K.  Deserter  Inf ormat ion--concerned  solely  with  deserter 

related  inlormation.  E.xamples  are:  deserter  claims  and 

actions . 

L.  Logist ics-^ includes  information  relative  to  movement  of 
supplies  (other  than  armament)  and  equipment  availability. 

M.  Unique  Act ions--includes  information  about  those  actions 
which  are  so  out  of  the  ordinary  that  they  command  special 
attention.  An  example  is  the  entry  of  the  Commanding 
General  into  the  area. 

N.  Contents  of  Documents--concerned  with  information  derived 
from  written  materials.  An  example  is  information 
derived  from  captured  documents. 

O.  Civilian -Mil i tary  Relat ion shins — solely  concerned  with 
information  regarding  the  interactions  and  relationships 
of  civilians  and  the  military.  An  example  is  the  coopera- 
tion between  the  population  and  the  aggressor. 
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Class  Definitions 


1.  Composition  of  Units — the  names  and  composition  of  aggressor 
units  and  temporary  attachments  of  the  various  types  of 
units  (e.g.,  mechanized,  artillery,  engineering). 

2.  Markings  and  Insignia — the  markings  or  insignia  of  all 
the  various  types  of  military  units  and  items. 

3.  Position  of  Forces  and  Eguipment--the  specific  location  of 
all  types  of  units,  .fortifications,  and  installations. 

4.  Strength  of  Units — the  absolute  and  relative  strength  of 
units  including  number  and  type  of  equipment  and  number 
of  men . 

5.  Replacements — the  receipt,  source,  and  deployment  of 
personnel  replacements. 

6.  Quantitative  Indications — numerical  or  related  quantitative 

statements.  Examples  of  such  statements  are:  30^  of, 

little,  rest  of,  many,  per  day,  a high  percentage  of, 
poorly,  to  the  fullest  and  easily. 

7.  Weapons  and  Equipment — includes  information  about  types, 
characteristics,  sizes,  issue,  supply,  maintenance , and 
deployment  of  weapons  and  equipment. 

8.  Analysis — concerns  detailed  analyses  of  various  types. 
Included  in  this  class  is  information  such  as  analyses  of  : 
reports,  data,  aggressor  actions,  etc. 

9.  Characterization  of  .Missions  — the  nature  of  various  missions 
(e.g.,  defensive,  support,  reconnai sance ) and  directions, 
requests  and  statements  relative  to  current  and  planned 
action . 

10.  Sabotage — sabotage  or  the  effects  of  sabotage  efforts 
on  the  part  of  either  friendly  or  aggressor  forces. 

11.  Actual  and  Threatened  Coni act--threatened  and  actual  contact 
and  potential  contact  that  did  not  materialize.  E.xamples 

of  the  typos  of  information  included  in  this  class  are; 
exchange  of  small  arms  fire,  and  whether  or  not  contact  is 
made . 
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12.  Strategy  and  Tactical  Doctrines--  statements  about  aggressor 
tactical  doctrine  (such  as,  emphasis  on  maneuverability). 

13.  Preparations  and  Readine.ss--contains  information  concerning 
capability  and  preparedness. 

14.  Movemetvt-- concerned  with  movement  of  all  types  of  units. 

Included  in  this  class  is  such  information  as:  direction 

and  location  of  movement  of  convoys  of  various  types. 

15.  Chrono log --concerned  with  all  statements  which  refer  to  time 
of  day,  date,  or  duration.  This  class  includes  information 
about  specific  times  and  dates  as  well  as  general  statements 
about  time  (such  as,  in  the  past,  and  nightly). 

16.  Information  about  Of f icers--includes  information  relevant  to 
aggressor  officers.  This  includes  names  and  backgrounds 

of  individual  officers  as  well  as  other  assorted  information 
(such  as  disturbances  in  an  officer's  personal  life). 

17.  Competence  of  Personnel  and  Equipment--training  and 
experience  of  personnel  and  current  capabilities  and  train- 
ing of  units  and  individuals. 

18.  Mental  and  Physical  Condition  of  Troops--morale,  indica- 
.tions  of  morale  and  health  conditions  of  both  the  aggressors 

and  civilians. 

19.  C ivilian  Military  Rel at ions--contact  and  relations  between 
the  military  and  the  civilian  population.  Depending  upon 
the  nature  of  the  military-civilian  contact,  this  informa- 
tion may  or  may  not  also  be  classified  under  the  factor 
Civilian-Military  Relationships  (factor  0).  For  example, 
"population  cooperates  with  aggressor,"  would  be  included 
in  this  class  and  would  also  be  included  under  the 
Civilian-Military  Relationships.  However,  "patrol  questions 
residents,"  would  be  classified  in  both  this  class  and  in 
the  factor  Type  of  .Activity  (factor  C). 

20.  Sources /Mot hods  of  Obtaining  In  Format i on --the  sources  of 
information  and  the  methods  of  obtaining  information.  For 
example,  interrogation  of  an  aggressor  sabotage  team, 
deserter  l-)ased  reports,  sightings  of  vehicles  and  enemy 
activity  and  conversations  that  were  overheard. 

21.  Informer  Source-Mi  1 i tary--dcals  with  intelligence  sources 
such  as  d('serters  and  POWs. 
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22.  Informor  Soiirco-Non-M i 1 itary — deals  with  civilian 
Intelligence  soui'ces. 

23.  High  R("11nhtltty  Source — intelligence  sources  such  as: 
radar  based  reports,  ground  surveillance  equipment  data, 
and  reports  from  U.S.  units. 

24.  Low  Rellahil Itv  Source — intelligence  sources  such  as: 
civilian  radio  broadcasts  and  rumor. 

25.  Conditional  Phrases--phrases  such  as:  since,  only,  when, 

as  a result  of,  after,  and  indicates. 

26.  Unknown  Information--ref ers  to  informaticm  that  lacks  some 
relevant  facts  (such  as  unit  designations,  destination, 
mission,  or  location). 

27.  Other  ---i  terns  that  do  not  seem  to  fit  into  any  other  class. 

Examples  of  this  are:  negation,  references  to  causes 

(e.g.,  political  unrest), and  unusual  words  (e.g.,  commercially 
and  telephone). 


IV . MODEL  CATEGORIZATION  OF  PHRASES 


This  compendium  of  categorized  phrases  is  an  example  of  the 
proper  use  of  the  system.  Key  phrases  were  derived  from  a set  of 
battlefield  messages  and  classified  by  an  expert  in  the  use  of 
this  system.  A* complete  list  of  all  types  of  information  that 
might  possibly  be  stored  in  a given  category  is  not  included. 

Rather,  an  example  of  what  resulted  from  the  application  of  the 
system  to  one  set  of  messages  is  included.  This  model  should  be 
used  to  clarify,  when  necessary,  the  type  of  information  that 
may  be  included  in  a given  category. 

The  model  was  developed  as  part  of  a linguistic  analysis  of 
battlefield  language.  The  messages  were  broken  up  into  two 
different  types  of  meaningful  phrases.  Both  types  are  seen  in  the 
pages  that  follow.  To  clarify  the  interpretation  of  these  two 
types  of  phrases,  consider  these  examples: 

type  1 - in  the  broadcast 

type  2 - promise  (broadcasts,  safe  passage  and 
assylum) 

The  first  type  of  phrase  is  read  as  usual.  The  second  type  of 
ph  rase  must  have  the  words  rearranged  before  being  read,  as  follows: 
"broadcasts  promise  safe  passage  and  assylum."  The  pattern  can  be 
Skplained  using  a shorthand  notation  substituting  letters  for  words. 

A phrase  written  in  the  form  "a(b,c)"  is  read  "bac."  In  general, 
the  whole  phrase  must  be  considered  to  understand  the  classification. 

Ah  understanding  of  the  reasons  for  the  use  of  two  types  of 
phrases  is  not  necessary  for  the  use  of  the  system.  Briefly  described, 
the  Second  type  of  phrase  usually  starts  with  a word  that  modifies 
the  Words  within  the  parentheses,  while  the  nature  of  the  first  typo 
Bt  jihrase  is  more  obscure. 
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Factor  A.  Movement  and  Capability  for  Movement 


i 

■ V, 


Class 

1.  Composition  of  Units  by  Type  and  by  Designation 

is  composed  of  (convoy,  15  half  ton  trucks) 
has  (“train,  25  cars)  ' • 

are  in  (amphibious  tanks,  the  force) 

is  loaded  with  (train,  combat  supplies  and  ammunition) 
carries  (train,  closed  boxcars) 
is  (apparent  cargo,  troop  replacements) 
is  attached  to  (unknown  mech  unit,  IF  div) 
is  attached  to  (an  element  of  the  eng  bn,  IF  mech  div) 
of  the  IF  div 

within  the  34th.  and  35th.  mech  div 


2.  Markings  and  Insignia 

wear  (aggressor  enlisted  men,  35  mech  div  insignia) 

have  (14  trucks,  16  CA.4  bumper  markings) 

are  (bumper  markings,  210) 

are  marked  (10  heavy  tanks,  3-10) 

have  (trucks,  34  mech  div  markings) 

bear  (patrols,  IF  mech  insignia) 

are  designated  (bridges.  Tango  2) 

have  (trucks,  93  armored  regt  markings) 


3.  Position  of  Forces  and  Equipment 
is  in  (IF  mech  div,  area) 

are  throughout  (personnel  and  vehicles,  the  area) 

is  in  (IF  mech  div,  this  location) 

are  at  (pontoon  sections,  vie  UR  048900) 


4.  Strength  of  Units 

is  (the  personnel  strength  of  the  35  mech  div,  70-80% 
is  (strength  of  547  mech  regt,  78%  of  TOSiE) 


7.  Weapons  and  Equipment 

are  heavy  (pon  sections) 

support  (sections,  GO  tons) 

is  (unit,  amphibious) 

deploys  (210  rgt  pontoon  sections) 

are  laid  (pun loons) 

are  equipped  with  (some  tank  units,  fording  equipment) 
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TOScE) 


Factor  A.  Movomont  and  Capability  for  Movement 


(continued) 


Analysis 

analyzes  (corps,  aggressor  ammunition  movement) 

Characterization  of  Missions 

is  (mission,  to  support  major  river  crossing) 

IF  mech  div  requests  that 

Sabotage 

is  sabotaged  (aggressor  rail  system) 

Actual  and  Threatened  Contact 

did  not  cross  (aggressor  aircraft,  border) 
has  mission  of  attacking  (rgt  20  mech  div) 

Strategy  and  Tactical  Doctrine 

stresses  (aggressor  tactical  doctrine,  speed  and  shock  action) 
Preparations  and  Readiness 

prepare  (aggressor  forces  of  35  mech  div,  bridges) 

Movement 

move  (trucks,  NE) 

moves  ( train,  south ) 

move  (4  large  tk  trucks,  north) 

move  (troops,  SE) 

move  to  (regt,  fwd  assembly  area) 

moves  (aggressor  unit,  into  territory) 

occurs  (crossing  into  friendly  territory) 

occurs  (forward  displacements) 

comes  in  (traffic) 

heads  (convoy,  NE  VIC  US  155095) 

leaves  (train,  AUE  (US  3706)) 

heads  (train,  SW  VIC  UR  115955) 

pass  through  (aggressor  units,  town  of  Pfaffengrun) 
pass  through  (two  groups  of  wheeled  vehicles) 
pass  (one  large  group  tks  or  SP  guns) 
is  indicated  (a  large  movement  of  troops) 
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visits  (G/A  Mikhaylov,  maneuver  area  of  34  mech  div) 

17.  Competence  of  Personnel  and  Equipment 

is  ranked  as  (IF  mech  div,  excellent) 
is  trained  (IF  mech  div)  • 

20.  Sources/Methods  of  Obtaining  Information 

have  been  sighted  (mech  units) 

are  sighted  (both  support  and  attack  type  aircraft) 
is  from  (deserter,  2/25  mech  regt  35  mech  div) 
comes  (deserter,  to  our  lines) 

26.  Unknown  Information 

is  unknown  (destination) 

27.  Other 

by  air 


I 
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Factor  D..  Numbers,  Trends  and  Source 


Class 

1.  Composition  of  Units  by  Type  and  by  Designation 

is  composed  of  (convoy,  15  half  ton  trucks) 

has  (train,  25  cars) 

consists  of  (patrol,  20  men) 

are  used  (400  additional  civilian  workers) 

receive  (aggressor  units,  300  men) 


2.  Markings  and  Insignia 

have  (14  trucks,  16  CAA  bumper  markings) 
are  marked  (10  heavy  tanks,  3-10) 


3.  Position  of  Forces  and  Equipment 

are  in  (major  aggressor  units,  the  following  areas) 

are  in  (major  aggressor  units,  Schoneck) 

are  in  (2  aggressor  bns,  Schoneck) 

is  in  (large  storage  area,  VIC  UR  032981) 


5.  Replacements 

receives  (16  CAA,  160  men) 


6.  Quantitative  Indications 

percent  of 
a large  number  of 
fewer  than 
many  of  the 
per  day 
little 

large  volume  of 


7.  Weapons  and  Equipment 

are  new  (weapons) 
is  defective  (equipment) 


11.  Actual  and  Threatened  Contact 

is  made  with  (significant  contact,  friendly  units) 
is  minor  (contact) 
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Factor  B.  I'mmbors.  Trends  and  Source 

(continued) 

Class 

14.  Movement 

pass  (two  groups  of  wheeled  vehicles) 
pass  (one  large  group  tks  or  SP  guns) 
is  indicated  (a  large  movement ' of  troops) 


15.  Chronology 

nightly 
continually 
continuing  for  2 hour 
for  the  past  96  days 
beginning  130240  sep 


18.  Mental  and  Physical  Condition  of  Troops 
arrested  (MPs,  12  aggressor  soldiers) 


20.  Sources/Methods  of  Obtaining  Information 
have  been  sighted  (mech  units) 

are  sighted  (both  support  and  attack  type  aircraft) 
is  from  (deserter,  2/25  mech  regt  35  mech  div) 
is  from  (deterter,  210  eng  regt) 


1.  Composition  of  Units  by  Type  and  by  Designation 

is  composed  of  (convoy,  15  half  ton  trucks) 
consists  of  (patrol,  20  men) 


2.  Markings  and  Insignia 

bear  (patrols,  IF  mech  insignia) 

3.  Position  of  Forces  and  Equipment 

« 

is  in  (jamming  unit,  IF  mech  div  area) 

4.  Strength  of  Units 

is  implemented  (rationing) 


7.  Weapons  and  Equipment 

is  fired  (weapon) 

had  (the  IF  arty  div,  misfires) 

use  (aggressor  troops,  weapons) 

maintain  (aggressor  troops,  weapons) 

deploys  (210  regt  . pontoon  sections) 

are  equipped  w'ith  (some  tank  units,  fording  equipment) 


8.  Analysis 

analyzes  (corps,  aggressor  ammunition  movement) 
has  been  analyzed  (recent  history  of  the  16  CAA) 


9.  Characterization  of  Missions 

is  (mission,  to  support  major  river  crossing) 
is  (mission,  infared  photography) 
is  being  carried  out 


10.  Sabotage 

is  sabotage  (the  aggressor  rail  system) 

performs  sabotage  (dissidents  within  the  official  party) 
destroys  (aggr('ssor  sabotage,  POL  supplies) 
inf  il  trait's  (aggresstir  sabotagt*  team,  US  lines) 
creatt's  (aggrt'ssor  sabotage  team,  confusion) 
jams  ( aggrt'ssor  , transmission) 


Factor  C.  Typo  of  Activity 


(continued) 

Class 

11.  Actual  and  Threatened  Contact 

has  mission  of  attacking  (rgt  20  mech  div) 
is  made  with  (significant  contact,  friendly  unit) 
is  minor  (contact) 
is.  made  (contact) 

is  made  (no  contact)  • ... 

have  contact  with  (US  units,  aggressor  patrols) 
withdrew  (enemy  patrols) 

. disengaged  (aggressor) 
occurs  (combat) 

occurs  (a  short  (3  min)  exchange  of  small  arms  fire) 
provokes  (US,  aggressor) 


13.  Preparations  and  Readiness 

did  emplacement  of  (work  force,  aggressor  mine  fields) 

prepare  (civilian  workers,  fortifications) 

are  being  prepared  (heavy  gun  emplacements) 

are  prepared  (positions) 

occurs  (preparation) 

prepare  (aggressor  forces  of  35  mech  div,  bridges) 


14.  Movement 

are  major  (logistical  movements) 


15-  Chronology 

during  contact 

for  the  duration  of  exercise 


16.  Information  about  Officers 

reviews  (CG,  progress  of  maneuvers) 


18.  Mental  and  Physical  Condition  of  Troops 
arrested  (MPs,  12  aggressor  soldiers) 

19-  Civilian-Military  Relations 

relocates  (unit,  some  civilian  residents  of  Warschitz  and 
Lcubetha ) 

questions  (patrol,  residcnt.s) 


ICO 


r 


li 


Factor  D.  Training 


Morale  and  Attitudes 


Class 

5.  Replacements 

are  trained  (replacements) 

are  from  (replacements,  satellite  nations) 


7.  Weapons  and  Equipment 

are  superior  (US  forces) 


16.  Information  about  Officers 

reviews  (CG,  progress  of  maneuvers) 
graduated  from  (G/D  Borodin,  Alta  Komando  College) 
are  from  (eng  officers,  16  CAA  staff) 
is  depressed  (Turgenev) 

is  well  versed  in  (G/D  Borodin,  urban  conflict) 


17.  Competence  of  Personnel  and  Equipment 

are  trained  in 
is  trained  (IF  mech  div) 

. is  rated  as  (IF  mech  div,  excellent) 
have  (replacements , technical  qualifications) 
are  competentin  (most  officers  and  enlisted  men,  individual 
combat ) 

are  competent  in  (most  officers  and  enlisted  men,  weapons 
operation ) 

are  competent  in  (most  officers  and  enlisted  men,  CBR  warfare) 
are  able  (generals  and  marshalls) 

exists  (capability  for  large  scale  operational  support) 

proves  (the  exercise,  some  major  new  concepts) 

have  (officers,  high  level  staff  experience) 

have  graduated  from  (senior  NCOs , aggressor,  NCO  academy) 

are  trained  in  (units,  urban  combat) 

are  apparent  (training  and  discipline) 


Factor  D.  Training,  Morale  and  Attitudes 


(continued ) 


Class 

l8.  Mental  and  Physical  Condition  of  Troops 

arrested  (MPs,  12  aggressor  soldiers) 

broke  out  (a  fight) 

may  cause  (integration,  problems) 

are  confused  (people) 

is  declared  (martial  law) 

is  good  (morale) 

have  been  disclosed  (several  courts  martial) 
has  increased  (abuse  pilferage) 


19.  Civilian-Military  Relations 

cooperates  with  (population,  aggressor) 
relocates  (unit,  some  civilian  residents  of  Warschitz 
and  Lcubetha) 

cooperate  with  (civilians,  military) 

cooperate  with  (political  leaders,  aggressor  military) 
is  apprehensive  of  (population,  nuclear  weapons) 


27.  Other 

due  to  the  recent  political  unrest 
because  of  political  beliefs 
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Factor  E.  Communications 


Class 

10.  Sabotage 

jams  (aggressor, transmissions) 


13.  Preparations  and  Readiness 

alerts  (signal, IF  mech  div) 


18.  Mental  and  Physical  Condition  of  Troops 

promise  (broadcasts,  safe  passage  and  asylum) 
is  poor  (aggressor  communication  discipline) 


20.  Sources/Methods  of  Obtaining  Information 

were  overheard  (conversations) 

are  from  (reports,  aggressor  homeland) 

intercepts  (unit,  radio  traffic) 

is  intercepted  (signal) 

are  monitored  (radio  broadcasts) 


26.  Unknown  Information 

is  unintelligible  (message) 


27.  Other 


by  telephone 


Factor  F.  Armament  Preparation 


Class 

7.  Weapons  and  Equipment 

are  in  (aggressor-mine  fields,  patterns) 
are  bunkered  (positions) 


13.  Preparations  and  Readiness 

did  emplacement  of  (work  force,  aggressor  mine  field) 
prepare  (civilian  workers,  fortification's) 
are  being  prepared  (heavy  gun  emplacements) 
are  prepared  (positions) 

prepare  (aggressor  forces  of  35  mech  div,  bridges) 

16.  Information  about  Officers 

supervise  (eng  officers,  emplacement) 


! 

I 


I 
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Class 

7.  Weapons  and  Equipment 


has  been  increased  (range  of  rifle) 

is  important  (new  rifle) 

is  known  about  (little,  new  rifle) 

are  similar  to  (weight  and  size,  US  M-16) 
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Factor  II.  Strength  and  Location  of  Units 


i 


r 


k 


Class 

1.  Composition  of  Units  by  Type  and  Designation 

is  composed  of  (IF  mech  div,  211  MDM  TK  RGT) 

is  composed  of  (IF  mech  div,  208  mech  rgt ) 

is  composed  of  (IF  mech  div,  211  mech  regt) 

is  composed  of  (IF  mech  div,  214  mech  rgt) 
is  in  (unit,  211  rgt) 

is  (apparent  cargo,  troop  replacements) 
consists  of  (patrol,  20  men) 
are  used  (400  additional  civilian  workers) 
receive  (aggressor  units,  300  men) 

2.  Markings  and  Insignia 

have  (14  trucks,  16  CAA  bumper  markings) 
are  marked  (10  heavy  tanks,  3-10) 

3.  Position  of  Forces  and  Equipment 

is  in  (IF  mech  div,  area) 

is  in  (54  mech  div  19  CAA,  vie  Auerbach) 

is  in  (deserter's  unit,  UR  037847) 

is  in  (unit,  vie  Neustadt) 

is  in  (div,  position) 

are  in  (major  aggressor  units,  the  following  areas) 

. are  (aggressor  units,  further  north) 
are  in  (major  aggressor  units,  Schoneck) 
is  in  (jamming  unit,  IF  mech  div  area) 
are  in  (two  agressor  bns,  Schoneck) 
are  throughout  (personnel  an'  vehicles,  the  area) 
is  in  (IF  mech  div,  this  location) 
is  north  of  (aggressor,  three  corps  area) 
is  at  (engineering  equipment,  this  vicinity) 
are  at  (pontoon  sections,  vicinity  UR  048900) 
is  in  (large  storage  area,  vie  UR  032981) 

4.  Strength  of  Units 

is  (the  personnel  strength  of  the  35  mech  ■div,70-80'c  TO&E) 

is  unit  strength,  83%  of  T0&.E) 

is  (strength  of  547  mech  regt,  78%  of  TO&E) 

requires  (strength,  attention) 

is  implemented  (rationing) 

are  strong  (major  or  subordinate  units) 

are  reduced  (unit  strengths) 


J 
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Factor  H. . Strength  and  Location  of  Unit 

(continued ) 


Class 

5.  Replacements 

receives  (16  C\A,  160  men) 
receive  (units,  replacements) 
is  requisitioning  (16  CAA,  replacements) 
receive  (aggressor  units,  replacements) 
have  been  replaced  (personnel) 

will  be  integrated  into  (replacements,  16  CAA  units) 


6.  Quantitative  Indications 


a highly  resistant  strain  of 


7.  Weapons  and  Equipment 

are  equipped  with  (some  tank  units,  fording  equipment) 


8.  Analysis 

is  analyzed  (aggressor  strength) 


14.  Movement 

move  (4  large  tk  trucks,  north) 

pass  (one  large  group  of  tks  or  SP  guns) 


16.  Information  about  Officers 

are  in  (generals  and  marshalls,  the  aggressor  army) 

are  in  (officers,  aggressor  ground  forces) 

is  assigned  to  (aggressor  major,  16  CAA  general  staff) 

is  (I.R.  Presnosff,  BN  CDR) 

is  (Col  V Weir,  Rgt  CDR) 

was  replaced  (deserter's  unit's  CDR) 

is  CG  of  (G/A  Mikhaylor,  D.,  19  CAA) 


18.  Mental  and  Physical  Condition  of  Troops 

have  (hospitals,  high  level  of  activity) 
infects  (influenza,  several  units) 


Factor  H.  Strength  and  Location  of  Unit 

(continued) 


Class 

20.  Source/Method  of  Obtaining  Information 

are  sighted  (both  support  and  attack  type  aircraft) 


26.  Unknown  Information 


does  not  know  (deserter,  location  of  unit) 

are  not  determined  (unit  designation^  and  strength) 

are  not  reported  (hospital  locations) 


Factor  I , 


Plans  and  Expectat ions 


Class 

5.  Replacements 

is  requisitioning  (16  CAA , replacements) 


9.  Characterizations  of  Missions 

have  (military  uses,  priority) 

is  (mission,  to  support  major  river  crossing) 

is  on  defensive '( 16  CAA) 

will  be  carried  out  (assigned  missions) 

has  mission  of 

intend  to 

purpose  is  to 

for  purpose  of 

with  purpose 

their  mission  is 

the  16  CAA  authorizes 

IF  mech  DIV  requests  that 

signal  instructs 

enemy  permits 


11.  Actual  and  Threatened  Contact 

has  mission  of  attacking  (rgt  20  mech  div) 
exists  (a  threat  to  the  border) 

12.  Strategy  and  Tactical  Doctrine 

stresses  (aggressor  tactical  doctrine,  speed  and  shock 
act  ion  ) 

is  ramified  (tactical  doctrine) 

emphasizes  (aggressor  tactical  doctrine,  fire  and  maneuver) 
is  emphasized  (combat  power  of  the  aggressor) 


13.  Preparations  and  Readiness 

are  being  prepared  (heavy  gun  emplacements) 

are  prepared  (positions) 

occurs  (preparation) 

alerts  (signal,  IF  mech  div) 

is  prepared  (div) 
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Factor  I.  Plans  and  Expectations 


(continued) 


Class 

14.  Movement 

move  (aggressor  unit,  into  positions) 


15.  Chronology 

soon 

immediately 
beginning  130240  Sep 
for  early  Sep 

16.  Information  about  Officers 

is  on  (CG  G.S.  Kuvov , offensive) 

emphasizes  (G/0  Borodin,  the  use  of  infantry) 


17.  Competence  of  Personnel  and  Equipment 

are  competent  in  (most  officers  and  enlisted  men,  CBR  warfare; 
exists  (capability  for  large  scale  operational  support) 


18.  Mental  and  Physical  Condition  of  Troops 

may  cause  (integration,  problems)  j 

promise  (broadcasts,  safe  passage  and  asylum)  J 
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Factor  J.  Extra-Military  Infornuition  Sources 

Class 

18.  Mental  and  Physical  Condition  of  Troops 

promise  (broadcasts,  safe  passage  and  asylum) 

20.  Sources/Methods  of  Obtaining  Information 

were  overheard  (conversations) 

were  interrogated  (civilian  line  crossers) 

are  from  (reliable  civilian  informers,  East  Planen) 

22.  Information  Source-Non  Military  . 

civilians  report 

civilian  reports  from  34  div  area  indicate 
friendly  civilians  indicate 
civilians  indicate 
friendly  civilians  overhear 
friendly. civilians  verified 
civilian  reports  indicate 
reliable  civilian  informers  report 
civilians  sight 

rumers  continue  within  the  civilian  community 


‘27.  Other 

in  the  broadcast 


I 
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16.  Information  about  Officers 


was  replaced  (deserter's  unit's  CDR) 

A 20.  Sources/Methods  of  Obtaining  Information 

is  from  (deserter,  2125  mech  regt  35*mech  div) 
has  instructed  (commander,  deserter) 
j ' is  from  (deserter,  210  eng  regt) 

, bases  claim  on  (deserter,  observation) 

comes  (deserter,  to  our  lines) 
picks  up  (patrol,  deserter) 


21.  Information  Source — Military 

deserter  states  that 
deserter  stated 
deserter  claims  that 
deserter  reports 
deserter  indicated 


26.  Unknown  Information 


noi  knnw  C rlPKPT'tpT  1r>r>ntinn  of 


Factor  L.  Logistics 


r 


m- 


Class 

1.  Composition  of  Units 

is  loaded  with  (train,  combat  supplies  and  ammunition) 
is  (apparent  cargo,  supplies) 

7,  Weapons  and  Equipment 

deploys  (210  regt . , pontoon  sections) 

10.  Sabotage 

is  sabotaged  (the  aggressor  rail  system) 

14.  Movement 

are  major  (logistical  movement) 

26.  Unknown  Information 

is  unknown  (destination) 
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Factor  M.  Unique  Actions 


Class 

7.  Weapons  and  Equipment 
employ  (units,  CBR) 

9.  Characterization  of  Missions 
' CG  IF  orders 

11.  Actual  and  Threatened  Contact 
. creates  (US,  international  incident)  . 

16.  Information  about  Officers 

visits  (G/A  Mikhaylov,  maneuver  area  of  34  mech  div) 
reviews  (CG,  progress  of  maneuvers) 
is  on  (CG  G.S.  Kurov,  offensive) 
is  CG  of  (G/A  Mikhaylov,  area  of  34  mech  div) 

17.  Competence  of  Personnel  and  Equipment 

are  competent  in  (most  officers  and  enlisted  men,  CBR  warfare) 
proves  (the  exercise,  some  major  new  concepts) 

20.  Sources/Methods  of  Obtaining  Information 
• has  instructed  (commander,  deserter) 


Factor  N. 


Contents  of  Documents 


Class 

5T  Analysis 

are  analyzed  (data  and  reports) 

are  Available  to  (data  and  reports,  G-2  3 corps) 


9.  Characterization  of  Missions 

aggressor  document  stresses  the  requirement  for 


20.  Sources/Methods  of  Obtaining  Information 

is  intercepted  (document) 
obtain  (intelligence  sources,  document) 
is  captured  (document) 
obtains  (civilian,  document) 


Factor  0.  Civilian  and  Military  Rel ations 


Preparations  and  Readiness 

prepare  (civilian  workers,  fortifications) 

Civilian-Military  Relations 

cooperates  with- (population , aggressor) 

relocates  (unit,  some  civilian  residents  of  Warschitz  and 
Leubetha) 

cooperates  with  (civilians,  military) 

cooperates  with  (political  leaders,  aggressor  military) 

questions  (patrol,  residents) 

are  within  (civilians,  IF  mech  area) 

is  apprehensive  of  (population,  nuclear  weapons) 


Sources/Methods  of  Obtaining  Information 

were  interrogated  (civilian  line  crossers) 
obtains  (civilian,  document) 

are  from  (reliable  civilian  informers.  East  Plauen) 


Informer  Source — Non-Military 


rumors  continue  within  the  civilian  community 


V.  MODEL:  ISONYMS  BY  CATEGORY 


The  user  of  this  system  should  be  alerted  to  the  imprecise 
use  of  words  in  the  battlefield  language.  Listed  here  are  words 
which  are  used  synonomously  within  categories.  This  list  was 
derived  from  the  messages  used  in  the  model  in  Section  IV  and 
is  not  exhaustive. 


Isonyms 


Category 


carries 

cargo 

wear 

bear 

have 

marked 

designated 

insignia 

deploy 

lay 

moves 

heads 

composed  of 
consists  of 

composed  of 
consists  of 

sighted 

observed 

captured 
picked  up 

loaded  with 
cargo 

intercepted 

obtained 

captured 

trained  in 
competent  in 
are  able 

due  to 
because 

intercepted 

monitored 


composed  of 
attached 
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Category 


H26 


Isonyms 


does  not  know 
are  not  determined 


4^ 

t" 

-i 

I 

t 


i 

4 


19 


112 

J22 

K21 


mission 

intends 

purpose 

stress 

emphasize 

reports  . 
indicates 

stated 

claims 

reports 

indicates 


180 


abuse 
academy 
action 
act ivity 

D18 

D17 

A12,  112 
H18 

additional 

HI 

aggressor  (see  enemy) 

A2,  A7,  A8,  AlO,  All,  A12 , A13,  A14, 
Bl,  B3,  B18,  C7,  C8,  CIO,  Cll,  C13, 
C18,  C20,  D17,  D18,  D19,  ElO,  E18, 
E20,  F7,  F13,  HI,  H3,  115,  H7,  H16, 
112,  114,  LIO,  019 

air 

A27 

aircraft 

All,  A20,  B20,  H20 

alert 

E13,  113 

ALTA  Kommando  College 

D16 

♦ammunition 

Al,  A7,  A8,  C8,  LI 

amphibious 

Al,  A7 

analysis,  analyze 

A7,  A8,  C8,  H7,  N8 

ap arent 

D19 

apprehensive 

D19,  019 

.area 

A3,  A14,  B3,  C20,  H3 , J22 , M16,  019 

armored 

A2 

army 

H16 

arrest 

B18.,  C18,  D18 

artillery 

C7 

assembly 

A14 

assign 

19,  H16 

asylum 

E18,  118,  J18 

attach  (see  compose) 

Al 

attack 

All,  A20,  B20,  Cll,  H20,  111 

attention 

H4 

Auerbach 

H3 

authority,  authorize 

19 

available 

N8 

battalion 

B3,  H3,  H16 

! bear  (see  insignia) 

A2,  C2 

beliefs 

D27 

j beginning 

B15,  115 

blackout 

C27 

.border 

All,  111 

Borodin 

D16,  116 

i boxcar 

Al 

bridges 

A2,  A13,  C13,  F13 

1 broadcast 

E18,  E20,  118,  J18,  J27 

' ‘broke  out  (see  occur) 

D18 

i bumper 

A2,  D2,  H2 

' bunkers 

F7 

capable,  capability 

D17,  117 

1 capture 

C20,  N20 
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car  (see  boxcar) 

cargo 

carries 

cause 

CBR 

CDR 

CG 

civilian 


claims 
col . 
combat 
commander 


communication  • 
community 
^competent 

compose  (see  attach) 


concents 

condition 

conflict 

confusion,  confused 


Al,  B1 
Al,  HI,  LI 
Al 

D18,  118 

117,  D17,  M7,  M17 
H16,  K16 

C16,  D16,  1116,  116,  M9,  M16 

C18,  C19,  D19,  F13,  HI,  J20,  J21,  J22, 

013.  019,  020,  022 


K21 

H16 

Al,  Cll,  112,  D17,  LI 

K20,  M20 


E18 

J22,  022 
D17,  117,  M17 

Al.  Bl.  Cl.  HI 


D17,  M17 

C27 

D16 

CIO,  D18  


deserter 

*designate 

destination 

destroy 


determined 

•discipline 

disclosed 

disengage 

displacement 

dissident 

division 


doctri  n(’ 


documen  t 
dural  ion 
East  IMauen 
earl 


A20,  B20,  C20,  113,  H17 , H26,  K3 , K16, 
K20,  K21,  K26,  M20 
A2 

H26,  026,  L26 

CIO 


1126 

D17,  E18 
D18 

Cll 

A14 

CIO 

Al  , A2,  A3,  A-1,  A9,  All,  AlO , A17, 

A20,  B20,  C3,  Cll,  C13,  D17,  E13,  F13, 
HI,  H3,  H4,  19,  111,  113,  J22 , K20, 

MIC 

A12.  112  


N9,  N20,  020 
Cl  5 

J20,  020 
115 


r 

I 


k 

r 

emphasize  (sec  stress) 

emplacement 

employ 

enemy  (see  atrttressor) 

112,  116 

C13,  F13,  113,  F16 
. M7 

19,  Cll 

engineering  (battalion,  regt, 

Al,  B20,  D16,  F16,  H3,  K20 

of  f icer ) 

enlisted  men 

A2,  D17,  117,  M17 

equip 

A7,  C7,  H7 

equipment  (defective, 

B7,  H3,  H7 

• 

engineering ) 

excellence 

A17,  D17 

exchange 

Cll 

exercise 

M17 

exists 

111,  117 

1 

k 

, experience 

C15,  D17 

fewer 

B6 

field 

C13,  F13 

ff 

. fight 

D18 

fire,  fired 

C7,  Cll,  C20,  112 

forces  (see  troops,  groundf orce) A1 , A13,  C13,  D7 , F13,  H16 

#» 

fording  equipment 

A7,  C7,  II7 

fort  if icat ion 

C13,  F13,  013 

friendly  (lines,  territory. 

unit) 

A14,  Bll,  Cll,  J22 

forward 

A14 

general  (common) 

H16 

General  (see  officer) 

D17,  H16 

good 

D18 

' 

graduate 

D16,  D17 

groundforces  (see  troops. 

H16 

forces) 

groups 

A14,  B14 

gun  (see  weapon) 

C13,  F13,  113 

heads  (see  moves) 

A14 

heavy 

A2,  A7,  B2,  C13,  F13,  113 

1 

' 

high,  highly 

H6;  D7,  HI 8 

history 

C8 

► 

homel and 

E20 

hospital 

H18,  1126 

' increase 

G7,  D18 

immediately 

115 

implement 

C4,  H4 

important 

G7 

incident 

Mil 

indicate 

A14,  B14,  J21,  J22,  K21 

individual 

1)17 

infantry 

116 

infect 

H18 

inf i 1 trate 

CIO 

— 
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influenza 

informer 

infrared 

insignia  (see  marking, 
identification) 

H18 

J20,  J22,  020 
C9 

A2,  C2 

instruct 

“l9.  KSO,  MiiO 

integration 

H5,  D18,  118 

intelligence 

C20,  N20 

intend 

19 

international 

Mil 

intercept 

C20,  E20,  N20 

•interrogate 

C20,  J20,  020 

■iam 

C3,  CIO,  ■ ElO,  113 

know , known 

G7.  H26,  K26 

Kurov 

116,  M16 

laid 

A7 

.large 

A14.  B3,  B6,  B14,  D17.  H3 . 1114.  117 

leader 

D19,  019 

leaves  (see  depart) 

A14 

Leubetha 

C19,  D19 

level  (see  training) 

D17 

line  crossers 

J20,  020 

lines 

A20,  CIO,  K20 

little 

B6,  G7 

logistic,  logistical 

C14,  L14 

locat ion 

A3,  H3,  H26,  K26 

loaded 

Al,  LI 

M16 

G7 

maintain 

C7 

Major  (both  rank  and  adjective) 

A9,  B3,  C9,  C14,  D17,  H3,  114,  H16 
L14.  M17 

maneuver 

A14,  C16,  D16,  112,  M16 

many 

B6 

marking  (see  insignia, 
identification ) 

A2,  B2,  H2 

.Marshalls  (see  officer) 

D17,  H16 

martial  law 

D18 

MOM 

HI 

mechanized 

Al,  A2,  A3,  A4,  A9 , A13,  A16,  A17, 
A20,  B20,  C2,  C3,  Cll,  C13,  C20,  D17 , 

* 

E13,  F13,  III,  II3,  114,  19,  111,  113, 

K20,  M16,  019 


men 

message 

Mikhaylov 

military 

mine  

minor 

monitor 

morale 

move,  movement 

MP 

Nations 

*NCO 

Neustadt 

new 

nightly 
19  CAA 

.North  , Northeast 

nuclear 

number 

obtain 

observat  ion 

occur 
of  f ensive 
official 

officer  (see  Generals 
IF  (division) 

operat ions 
operat ional 
orders 

overheard 

party  (see  unit) 

pass 

passage 

patrol ___ 

, patterns 
people 
per 

percentage 

permit 
' personnel 
Pf af  fcngrun 

photography 

pick  up  (s(?e  capture 

pi  1 forage 

POL 

political  


Bl,  B5,  Cl,  HI,  H5 
E20,  E26 
A16,  H16,  M16 
D19,  19,  019 

F7.  C13,  F13 

Bll,  Cll 

E20 

D18 

A7,  A8.  A14.  B14.  C8 , C14 , H14.I14,  L14 

~ B18,  C18,  D18 

D5 
D17 

H3 — 

B7,  G7,  M17 
B15 

H3,  H16 

A14,  H3,  H14  

D19,  019 
B6 

N20,  020 

C20,  K20 

A14,  Cll,  C13,  113 

116,  M16 
CIO 

, Marshalls ) D16  , D17  , F16.  H16  117,  M17 

Al,  A3,  A9,  Al7,  Ci , C3,  C7,  D17 , E13, 

HI.  H3,  19.  M9,  019  

D17 

D17,  117 
M9 

C20.  J20.  J22 

CIO 

A14,  B14,  H14 
E18,  J18 

A2,  Bl.  Cl,  C2.  Cll.  C19.  C20.  HI  . K2Q^^2_ 



D18 

B6 

B6  

19 

A3,  A4,  H3,  H4,  H5 
A14 

C9 

) C20,  K20 

D18 
CIO 

D19,  D27.  019 
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K 


t 

pontoon  (pon  sections) 
1 population 
poorly,  poor 
position 

A3,  A7,  C7,  II3,  L7 

D19,  019 

E18 

A14,  C13,  F7.F13,  H3 , 113,  114 

i 

power 

112 

Presnof f 

H16 

prepare 

A13,  C13, 

F13,  113,  013 

b- 

priority 

19 

problems 

D18,  118 

progress 

C16,  D16, 

M16 

1 

promise 

E18,  118, 

J18 

L 

proves 

D17 

provoke 

Cll 

•purpose 

19 

qualification 

D17 

radio 

C20,  E20 

rail 

AlO,  CIO, 

LIO 

:> 

«. 

ramified 

112 

1 .range 

G7 

tr 

rated 

D17 

, ration 

receive 

C4,  H4 
H5 

recent  C8,  D7 

reduced H4 


regiment 

A2,  A7,  All  '14,  A20,  B20,  C7,  Cll,  HI, 
H2,  H4,  H'  .1,  K20,  L7 

reliable 

J20,  J22, 

relocate 

C19,  D19 

replacement 

A1.D5,  D17,  E20,  HI,  H5,  1116,  R26,  15, 
J21,  J22,  K16,  K21  ‘ 

reports 

N8,  E20, 

J21,  J22,  K21,  H26 

request 

19 

requirement 

H4,  N9 

requisition 

H5,  15 

resident 

C19,  D19, 

019 

resistant 

H6 

review 

C16,  D16, 

M16 

rifle  (see  weapon) 

G7 

river 

A9,  C9,  19 

rumor 

J22,  022 

16  CAA 

B2,  B5,  C8,  D16,  H2 , H5,  H16 , 15,  19 

sabotage 

AlO,  CIO, 

C20,  LIO 

safe 

E18,  118, 

J18 

satellite 

D5 

Schoncck 

B3,  H3 

senior 

D17 

September 

115 

shock 

A12,  112 

short 

Cll 

sight,  sighted 

A20,  B20, 

C20,  H20,  J22 
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signal 

19,  E13,  E20,  113 

signif icant 

Cll 

similar 

G7 

size 

G7 

small  arms 

Cll  ’ - 

soldier 

B18,  C18,  D18 

soon 

115 

source 

C20,  N20 

South 

A14 

Southeast 

A14 

Southwest . 

A14 

speed 

A12,  112 

SP  gun 

A14,  H14 

staff 

D16,  D17,  H16 

storage 

B3,  H3 

strain 

H6 

strength 

A4,  H4,  H7,  H26 

stress  (see  emphasis) 

A12,  112,  N9 

strong 

H4 

subordinate 

H4 

superior 

D7  . 

supervise 

F16 

supplies  (see  cargo) 

Al,  LI 

support 

A7,  A20,  B20,  D17,  H20 , 19,  117 

system 

AlO,  CIO,  LIO 

3 corps 

H3,  N8 

34  mech  div 

Al,  A2,  A16,  M16 

35  mech  div 

A1,A2,  A13,  A20,  B20,  C13,  F13,  K20 

tactics 

A12,  112 

Tango  2 

A2 

tank 

Al,  A2,  A7,  C7,  H2,  H7 

team 

C20 

technical 

D17 

telephone 

E27 

terr i tory 

A14 

threat 

111 

throughout 

A3,  H3 

TK 

A14,  111,  1114 

TO&E 

A4  , H4 

ton 

Al,  A7,  Bl,  Cl 

town 

A14 

training 

D5 

traf  f i c 

A14,  C25,  E20 

train  (v<'hicle) 

Al,  Bl,  LI,  All 

training,  trained 

A17,  D17 

transmission 

CIO,  ElO 

troops  (see  groundforcos ) 

Al,  A14,  B14,  C7,  H2 

truck  (s«'e  vohi<;lo) 

Al.  A2.  AM.  Bl.  Cl.  H2.  IIll 
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unknown 

unrest 

urban 


026,  L26 
D27 

D16,  D17 


vehicle  (see  aircraft,  tank. 

A3,  A14,  B14,  H3 

. train,  truck) 

verify 

J22 

vicinity 

A3,  A14,  C20,  H3 

visit  (see  enter) 

A16,  M16 

volume 

B6 

warfare  (see  combat) 

D17,  117,  M17 

Worschitz 

C19,  D19,  019 

weapon 

B7,  C7,  D17,  D19,  019 

wear 

A2 

weight 

G7 

Weir 

H16 

wheeled  (see  vehicle) 

A14,  B14 

withdrew 

Cll 

work 

F13 

workforce  (see  labor) 

C13,  F13 

workers 


C13,  F13^  HI,  013 
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A Measure  of  Agreement  Among  Several  Raters 
with  Each  Making  Several  Assignments  


Part  of  the  method  given  by  Fleiss  (1971)  for  measuring  agreement 
among  several  judges,  each  making  only  one  assignment,  is  extended  to 
each  making  more  than  one  assignment.  The  new  method  applies  a cor- 
rection to  each  individual  message  to  account  for  the  possibility  of  guessing. 

Let  k = number  of  categories  available  for  assignment  to  a message 
n = number  of  judges 

nj  = number  of  judges  who  assigned  category  j 
a = number  of  categories  assigned  to  a^message  by  each  judge 

A raw  measure  of  agreement  of  assignments  to  a particular  message  is  given 
by: 

k 

S n.(n.-l)- 
j=l  J J 

P = 

an(n-l) 


For  a = 1,  this  agrees  with  Fleiss.  The  agreement,  corrected  for 
the  possibility  of  guessing,  is  given  by:  . 


P - P 

r 

1 - P 


where:  P is  the  P obtained  by  assuming  the  most  random  results  possible 
for  n judges,  i.  e.  , the  assignments  (categorizations)  are  most  evenly  divided 
among  the  k categories. 
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Example  I 


As  an  example,  let  k = 60  categories,  n = 15  judges,  and  a = 6 as- 
signments per  judge.  There  are  thus  15x6=  90  assignments  distributed 
among  the  60  categories  or  an  average  of  1.  5 assignments  per  category. 
Therefore,  for  the  most  even  distributions  some  categories  will  have  tw'o 
assignments  and  some  one.  Let  x be  the  number  of  categories  having  two 
assignments.  Then,  there  will  be  60  - x categories  with  one  assignment. 
The  total  number  of  assignments  is  2x  + 1 (60  -x)  = 90,  which  gives:  x = 30. 
Therefore,  the  most  random  assignment  would  give  30,  which  have  agree- 
ments of  two  judges  and  30  single  assignments. 

Since  for  n-  = 2,  n.(n.-  1)  = 2,  and  for  n.  = 1,  n.(n.-  1)  = 0,  P has 
30  twos  in  the  numerator.  ' i i r 

^ 30(2)  J_ 

r 6(15)(14)  21 


which  is  the  result  obtained  from  the  most  likely  random  assignment.  The 
corrected  measurement  of  agreement  would  then  be: 


A 


P 


1 


1 

21 


21P  - 1 
. 20 


1.  05  P - . 05 


for  the  .parameters  assumed  in  this  example.  If  such  a case  actually  occurred, 
A would  be  zero.  For  P = 1,  A would  be  1.  Suppose  that  for  a given  message, 
there  was  actually  one  category  on  which  all  15  judges  agreed,  two  categories 
on  which  12  agreed,  two  on  w'hich  5 agreed,  and  41  categories  chosen  by  only 
one  judge.  This  gives: 


15(14)+  2[12(11)]  + 2(5(4))  + 41(0)  494 

6(15)(14)  ■ 1260 


A = 1.  05(.  392)  - . 05  = . 362 

Suppose  for  another  message,  there  w.ere  two  categories  on  which  14 
judges  agreed,  three  on  which  11  agreed,  seven  on  which  3 agreed,  and  four 
on  which  2 agreed. 

P - 2[14(13)1  + 3111(10)1  t 7[3(2)|  t 4)2(1)]  _ 744  _ 

1260  ■ 1260  " ■ 


A = 1,  05(,  591)  - , 05  = .57 
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The  syntax  diagram  shown  on  the  following  pages  was  chosen 
as  a means  to  display  the  syntax  which  the  user  (intelligence 
analyst)  of  the  system  would  employ.  This  type  of  diagram  affords 
a concise  composition  of  syntax  involving  details,  alternatives 
and  iterations  which  is  rigorous  without  being  cumbersome. 

( 

The  basic  rule  in  reading  those  diagrams  is  that  any  path 
traced  along  the  forward  direction  of  the  arrows  will  produce  a 
I syntactically  correct  component  of  the  user  language.  Those 

items  enclosed  by  < > indicate  elements  which  are  e.xplained  by 

L the  words  bracketed,  i.e.,  not  further  diagramatically  defined. 


Words  which  are  entirely  underlined  must  be  completely  specified 
by  user  keyboard  entry.  Words  which  are  partially  underlined  can  be 


abbreviated  by  the  underlined  part  or  any  part  containing  at  least 
the  underlined  part.  Accordingly,  INPUT  can  be  represented  by  IN, 
INP,  or  INPUT  but  not  by  I or  INPUTS. 

The  maximum  number  of  iterations  through  a portion  of  the 

diagram,  n,  is  indicated  An  asterisk  indicates  that 

the  indicated  path  must  be  taken  at  least  once. 

Messages  are  supplied  to  the  system  for  storage  using  the  first 
syntax  diagram  shown,  i.e.,  INPUT.  For  each  message  input,  at  least 
the  class,  and  factor  information  and  the  message  itself  must  be  pro- 
vided. For  a further  syntactical  definition  of  the  class  and  factor 
(as  w'ell  as  the  other  identified)  see  those  respective  diagrams 
shown  later. 

In  making  an  inquiry,  the  system  user  would  use  the  INQUIRY 
syntax.  The  synta.x  was  designed  to  allow  a scenario  such  as  that 
which  follows.  The  analyst  prepares  an  initial  set  of  selection 
criteria  describing  the  classes  of  messages  he  would  like  to  examine, 
search  or  count.  To  do  this,  he  enters  only  INQ  ALL,  WITH  followed 
by  any  one  or  more  items  in  the  selection  description  (next  to  last 
diagram)  followed  by  one  of  the  inquiry  types.  Consider  the  example; 


INO  ALL  WITH  CL  A,  FAC  2,  CL  F ,FAC  13,  KF.Y  BRIDGE 
FR  5-17-79  TO  NOW,  COUNT 

This  represents  a request  to  count  all  messages  of  categories 
A2  or  F13  containing  the  word  BRIDGE,  entered  into  the  system 
after  May  17,  1979. 


>■ 


¥ 


TJ>^analyst  is  presented  with  the  count. 

If  the  count  is  large,  the  analyst  may  enter  another  inquiry 
without  the  word  ALL  by  adding  information  to  the  selection  criteria. 
This  action  will  reduce  the  number  of  messages  to  be  visually  in- 
spected. For  example: 

INQ  FROM  "SENDERS  NAME",  -COUNT 

will  consider  only  those  messages  which  met  the  prior  selections 
criteria  and  also  have  the  property  that  they  wer  - sent  from  some 
specific  sender  (individual  or  organization  by  name). 

If  the  resultant  count  is  sufficiently  low  so  that  the  user 
would  like  to  see  a display  of  all  of  them,  he  then  enters: 

INQ  DISPLAY 

or  INQ  DISPLAY  F (first  page)  followed  by 
INQ  DISPLAY  N (next  page) 


INPUT 


Function : 
De  f au 1 1 : 


provides  message  information  to  system 
none 


MAINTENANCE 


MAINTENANCE 


Function:  controls  system  use 
Default;  no  users 


I NQU I RY 


INQUIRY 


selection  spec  > 


■»- < inquiry  type  spec  > 


// 


// 


Function:  asks  for  information  about  a set  of  messages.  If  the  word 

AIiL  is  used,  then  apply  the  < selection  spec  > to  all  messages 
from  the  set.  Otherwise,  apply  the  < selection  spec  > to  the 
last  formed  set  of  messages.  The  information  desired  is 
f specified  by  the  <inquiry  type  spec  >. 

Default;  normal  configuration. 


I 
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Function;  present  inquiry  results  to  user 
Default;  normal  configuration 


j 


•ARCHWES 


Function : 
Default : 


■COUNT- 


■SHOW 


•DETERMINE 


t: 


TERMINALS 


J 


INQUIRY- 


• ADD  term! na I - 
•DROP  term! na I 
•SECURITY  log 


DISPLAY 


user  training 
normal  configuration 


3 


< CLASS  SPEC  > 


< class  no  > 


Function;  specifies  one  or  more  of  the  27  classes  into  which 
battlefield  messages  may  be  assigned 
Default: . all  classes 


< FACTOR  SPEC  > : : = 


FACTOR 


< factor  no.  > 


Function:  specifies  one  or  more  of  the  15  factors  into  which 

battlefield  messages  may  be  assigned 
Default:  all  factors 


< SECURITY  SPEC  > : 


►SECURllY- 


< security  > 


< securi ty  > : 


UNCLASSIFIED- 


CONFIDENTIAL 


SECRET. 


TOP  SECRET 


Function:  defines  either  (1)  security  level  of  a message,  or 

(2)  highest  authorized  security  level  for  a terminal 
Default:  unclassified  security 


ENCODING  SPEC 


-HEADER 


•ENCODED 


• ENCODED 


Function:  specifies  whether  or  not  header  and/or  text  is  input  and 

stored  in  plain  or  encoded  form. 

Default:  header  plain,  text  plain 


MESSAGE  PRIORITY  SPEC 


-MESSAGE  PRIORITY 


< PRIORITY  > 


PRIORITY  ::= 


EMERGENCY  ACTION  MSG 


OPERATIONAL  IMMED 


PRIORITY 


ROUT  1 NE 


Funtion:  specifies  the  priority/criticality  of  a message 

Default:  message  priority=  routine 


MESSAGE  RELIABILITY  SPEC  ;:= 


* MESSAGE  RELIABILITY 


< re  I i ab i I i ly  > 


< reliability  > : : = 


CERTAIN  

CONFIRMED  

UNKNOWN  ^ 

SINGLE  SOURCE 
UNCONF IKMLD  — 


spocifioH  the  rol i abi 1 ity/cortainty  of  a message 
message*  reliability=  unknown 


Function : 
Default : 


ADD/REASSIGN  TERMINAL  SPEC  ;:= 


user 


code 


Function:  to  add  a terminal  to  the  network  or  to  reassign  a terminal 

by  redefining  its  location,  security  category  and/or 
applicable  user  codes 
t 

Default:  none 
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DATE  SPEC  : : = 


Function : 
Default : 


to  define  a time  period  over  which  messages  of  interest 
are  to  be  processed. 

from  start  of  system  to  today 


* 


f 
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< INQUIRY  TYPE  SPEC  > 


■<  selection  type  > 


DISPLAY 


-FIRST  -►page 


^►OETE  RHINE 


■<selection  type 


< logical  statement  > 


< logical  statement  > is  any  question  that  can  be  answered  "yes"  or  "no" 
analyzing  the  message's  text. 


KEYWORD  SPEC 


KEYWORD 


provides  the  capability  for  user  to  specify  one  or 
more  special  words  when  ho  would  like  to  count  or 
retrieve  stored  messages  containing  this  (these) 
words 


Default:  no  key  words 
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< SELECTION  SPEC  > 


<se I ecT I on  spec> 


ACl IVE 


ARCHIVE 


Each  <se  I (.'d  i on  spoo  defines  a set  of  messages,  each  message  having  all 
of  the  specified  conditions.  Sets  of  messages  can  be  combined  by  OTR 
(messages  in  either  of  both  sctsf  or  (messages  in  both  sets).  Ttius 
PR=  ROUTINE  AN[l  (SENDiP.  X OR  SENDER  Y) 
would  be  ttic  set  of  messarjes  of  priority  routine  sent  to  X or  Y 


